RAOC-VR-84-20 
In-Kob^o  Report 
January  t?84 


RELIABILITY  GROWTH  TESTING 
EFFECTIVENESS 


AD-A141  232 


Preston  R.  MacDiarmid  and  Seymour  F.  Morris 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 


DTIC 

ELECTE 

MM2  1  1984 

B 


ROME  AIR  DEVELOPMENT  CENTER 
Air  Force  Systems  Command 
Griffiss  Air  Force  Base,  NY  13441 


g  4  0  5  21  007 


Yjr,  v 


k  *. 

i  % 

h  .  • 


KO 


K  * 
S\- 

►■V 


i-« 


U§ 


This  report  has  been  reviewed  by  the  RADC  Public  Affairs  Office  (PA)  and 
is  releascble  to  the  National  Technical  Information  Service  (NTIS) .  At  M’IS 
it  will  be  releasable  to  the  general  public,  including  foreign  nations. 

RADC-TR-84-20  h3S  been  reviewed  and  is  approved  for  publication. 


Chief,  Systems  Reliability  &  Engineering  Branch 
Reliability  &  Compatibility  Division 


APPROVED: 


Acting  Chief,  Plans  Office 


.  S.  TUTHILL,  Colonel,  USAF 

hief,  Reliability  &  Compatibility  Division 


FOR  THE  COMMANDER: 


If  your  address  has  changed  or  is  you  wish  to  be  removed  from  the  RADC  mailing 
list,  or  if  the  addressee  is  no  longer  employed  by  your  organization,  please 
notify  RADC  (RBER)  Griff. iss  AFB  py  13441.  This  will  assist  us  in  maintaining 

a  current  mailing  list. 

Dc  rut  return  cop. ♦os  ov  ti.is  report  unlesn  contractual  obligations  or  notices 
on  a  specific  oo.-u  sent  ■**<?-puires  chat  it  be  returned. 


)  . 


I- 


\  J 


REPORT  DOCUMENTATION  PAGE 


RHAD  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


1.  RECIPIENT'S  CATALOG  NUMiCK 


4.  TITLE  (and  Subtitle) 

RELIABILITY  GROWTH  TESTING 
EFFECTIVENESS 


5.  TYPE  OF  REPORT  4  PERIOO  COVERED 

In-House  Report 


6.  PERFORMING  ORO.  REPORT  NUMBER 

N/A 


7.  AUTHOR*,) 

».  CONTRACT  OR  GRANT  NUMBERS 

Preston  R.  MacDiarmid 

Seymour  F.  Morris 

N/A 

».  PERFORMING  ORGANIZATION  NAME  AM  5  AOORESS 

10.  PROGRAM  ELEMENT.  PROJECT,  TAJK 
AREA  4  WORK  UNIT  NUMBERS 

Rome  Air  Development  Center  (RBER) 

62702F 

Griffiss  AFB  NY  13441 

23380289 

1  1.  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

12.  REPORT  OATS 

Rome  Air  Development  Center  (RBER) 

January  1984 

<  3.  NUMBER  OF  PAGES 

Griffiss  AFB  NY  13441 

172 

14.  MONITORING  AGENCY  NAME  4  ADORCSSf//  dlttaronl  tram  Controlling  Oltico) 

It.  SECURITY  CLASS,  (o t  thl ,  rrport) 

Same 

UNCLASSIFIED 

IS».  OEcLaSSI  FI  CATION/  DOWNGRADING 
n/aschedule 

14.  DISTRIBUTION  STATEMENT  (o  1  thim  Rtporl ) 

Approved  for  public  release;  distribution  unlimited 

17.  DISTRIBUTION  STATEMENT  (of  mbatrmcl  antorad  In  Block  20,  If  dlfl'arant  from  Rapor.) 

Same 

18.  SUPPLEMENTARY  NOTES 

None 

T9.  KEY  WOnC^  (Contlnva  on  tavotaa  j idw  It  nacaa  amry  and  tda^ulty  by  block 

Reliability 

Reliability  Growth 

Test  Analyze  and  Fix 

Duane 

Fi  x 

r 

20.  ABSTRACT  (Contlnum  on  tavaram  alda  It  nacaaaary  mid  idantlty  by  block  numbar ) 

This  in-house  report  documents  the  results  of  an  RADC  Systems  Reliability 
and  Engineering  Branch  in-house  study  on  reliability  growth  testing.  The 
study  involved  examination  of  DoD  policy  regarding  this  form  of  testing,  an 
extensive  literature  search  on  techniques  and  applications  as  well  as 

consultation  with  Air  Force  reliability  experts 

on  the  subject.  The  results 

aadress  a  general  overview  of  models  and  techniques  applied  with  particular 
attention  to  unique  approaches  found  in  the  literature.  Numerous  current- 

FORM  1i?1 

1  JAN  73  EDITION  OF  I  NOU  «3  IS  OBSOLETE 


_ UNCLASSIFIED _ 

SECURITY  CLASSIFICATION  of  THIS  PAGE  finivn  Data  Bntarad) 


UNCLASSIFIED 


««cuwTv_CLA>»)ftCATiOH  or  thu  Of  aiwm 

And  past  Air  Force  applications  are  cited  indicating  the  range  of  possibl 
approaches.  The  report  concludes  by  addressing  many  of  the  questions 
regarding  reliability  growth  testing  Repressed  by  those  skeptical  of  it. 


DTIC 

ELECTE 
MAY  2  1  1934 


B 


2) 


V - - - 

Accession  >' 

or  y 

NTIS 

CRA&I 

K 

DTIC 

TAB 

|  Unannounced 

□ 

j  Justl 

i 

f lent i un 

fiv _ 

,  Distribution/ 

1  Availability  Codes 

Avail 

and/or 

Dist 

Special  | 

Hi 

TABLE  OF  CONTENTS 


SECTION 

1,0  Objective 
2.0  Approach 

2.1  Issues 

3.0  Reliability  Growth  Testing  Terminology 

3.1  Reliability  Testing 

3.2  Growth  and  Failures 

3.3  Failure  Reporting  and  Corrective  Action  System 
(FRACAS) 

3.4  Reliability  Growth  Limiting  Values 

3.5  Reliability  Growth  in  Management 

3.6  Reliability  Growth  vs  Other  Reliability  Tasks 

3.7  No-Growth  Growth 


O  Q  Dnl  1  1  1  1  +  'll  r»AAI.l4-U  M  4  .>  A  A  A  ■*  A 

•  kj  i\ci  i  uu  i  i  »  vjr  ui  urun  n  iuiio 


4.0  DoD  Policy  on  Reliability  Growth  Testing 

4.1  Standards 

4.2  Development  Process 

4.2.1  Reliability  Development  Phases 

4.3  Tailoring  Tasks 

4.4  Direction 

4.4.1  DoD  Directive  5000.40:  "Reliability  and 
Maintainability"  (8  Jul  80) 


4.4.2  AFR  800-H8:  "Air  Force  Reliability  and 
Maintainability  Program"  (15  Jun  82) 


TABLE  OF  CONTENTS 


SECTION  PAGE 

4.4.3  MIL-STD-785B:  "Reliability  Programs  for  28 

Systems  and  Equipment  Development  and 
Production"  (15  Sep  8C) 

4.4.4  MIL-STD-781C:  "Reliability  Design  36 

Qualification  and  Production  Acceptance 

Tests:  Exponential  Distribution" 

(21  Oct  77)  (Currently  Under  Revision  to 


MI.-STD-781D,  See  paragraph  4.4.5) 

4.4.5  MIL-STD-781D  (31  Dec  80  Draft)  36 

4.4.6  MIL-STD-1635(EC):  "Reliability  Growth  37 

Testing"  (3  Feb  78) 

4.4.7  MIL-STD-2Cb8:  "Reliability  Development  39 

Testing"  (21  Mar  77) 

4.4.8  MIL-HDBK-189:  "Reliability  Growth  40 

Management"  (13  Feb  81) 

5.0  Reliability  Growth  Analysis  43 

5.1  Reliability  Growth  Model  Types  43 

5.2  Reliability  Growth  Models  47 

5.2.1  The  Duane  Model  47 

:  2.2  The  AMSAA  Model  53 

5.2.3  Duane -vs -AMSAA  Model  57 

5.2.4  Other  Models  59 

5.2.r  Nonrelevant  Failures  65 


TABLE  OF  CONTENTS 


•< 


SECTION 

6.0  Reliability  Growth  Management  Techniques 

6.1  Reliability  Growth  Test  or  Not 

6.2  Planning  for  Reliability  Growth 

6.2.1  Initial  Reliability 

6.2.2  The  Growth  Rate  (a) 

6.3  Reliability  Growth  Test  Time 

6.3.1  Reliability  Growth  Test  Time  Estimation 
for  a  System 

6.3.2  Allocating  Reliability  Growth  Test  Time  to 
Subsystems 

6.3.3  Test  Time  Example 

6.3.4  Planning  Test  Time 

6.4  The  Exponential  Law  for  the  Appearance  of 
Systematic  Failures 

6.5  Tracking  Techniques 

6.6  Confidence  Levels 

6.7  Cost  of  a  Growth  Program 

7.0  Re^ability  Growth  Application  Experience 

7.1  Current  Air  Force  Applications 

7.1.1  hAVE  CLEAR  (Formerly  SEEK  TALK) 

7.1.2  SACDIN 

7.1.3  AFSATCUM 


PAGE 


66 

67 

'!  A.  \ 

; 

72 

77 

77 

A 

83 

84 

86 

(fasweipi 

"  « *  ■  •  * 1 

.*  4  A  ■/  / 

■  >,v 

88 

r* 

93 

.vAAj 

«.  M 

93 

“m  ",  “d 

-  »  ^  * 

/./'A 

96 

99 

'/a*'!  • 

100 

rT^  V 

104 

104 

'.ava: 

105 

*  ^ 

106 

, "\.  s- 

106 

7.1.4  ,1TIDS 


107 


TABLE  01-  CONTENTS 


SECTION 

7.1.5  Simulator  SPO 

7.1.6  F-16 

7.1.7  Bl-B 

7.1.8  AMRAAM 

7.1.9  B-52  OAS 

7.1.10  AWACS 

7.1.11  AN/ARC-164 

7.2  Program  Application  Summary 
8.0  Conclusions 
8.1  Summary  of  Conclusions 
APPENDIX  A  Test  Time  Tables 

APPENDIX  B  Bibliography 


LIST  OF  TABLES 


TITLE 


MIL-STD-785B  Reliability  Test  Definitions 


DoD  Reliability  Related  Documents 
{Reliability  Test  Impact) 


Application  Matrix  for  Program  Phases 


Reliability  Phase  Terminology 


Prioritization  of  Standard  Reliability  lajks 


Task  Application  Guidelines  Based  on  Reliability 
Phase  Terminology 


Task  104  -  Failure  Reporting,  Analysis  and  Corrective 
Action  System  (FRACAS) 


Task  302  -  Reliability  Development/Growth  Test 
(RDGT)  Program 


Task  303  -  Reliability  Qualification  Test  (RQT)  Program  32 


MIL-STD-785B  Reliability  Growth  Application  Guidance 


MIL-STD-785B  Reliability  Qualification  Test 
Application  Guidance 


n-i.'.inu..  r* _ .a-u  / 1  icflur  \ 

t\ei  idui  my  uiuwuii  nuuc  r  wumpai  isuii  \\ionr iv/ ; 


Reliability  Growth  Study  System/Equipment  Descriptions  61 


Re1 i ability  Growth  Study  Equipment  Categories 


Reliability  Growth  Study:  Joint  Goodness  of  Fit  Analysis  63 
for  Airborne/Ground  and  In-Hous  'rield  Classifications 


Reliability  Growth  Study:  Model  Comparisons  by 
Equipment  Categories 


Reliability  Growtn  Rates  for  Electronic  Equipment  from  80 
Improvement  Programs  During  Service  Use 


Reliability  Growth  Rates  Observed  for  Different  Hardware  81 
Systems  in  Development  Tests 


,v  a 


'V V-.V-Wj %\-v 


iS'.'W 


•- 


•-1  ’  ; 


TABLE 


TITLE 


PAGE 


6-3  Examples  of  Reliability  Growth  Rates  Under  RIW  Programs  HP 

6-4  Comparison  of  Reliability  Growth  Rates  83 

6-5  Variations  of  Recormiended  Test  Times  Presented  in  the  84 

Literature 

6-6  Subsystems  and  Their  Required  MTBF'S  87 

6-7  Test  Time  In  Terms  of  Multiples  of  the  Required  MTPF  87 

6-8  Initial  Growth  Test  Data  88 

6-9  Rel ic bil ity  Attributes  and  Application  Levels  101 

6- 10  Reliability  Attribute  levels  for  a  Given  State  102 

7- 1  Air  Force  Reliability  Growth  Applications  105 

8-  i  Questions  Regarding  RUGT  Implementation  113 


LIST  OF  FIGURES 


TITLE 

Categorization  of  Defects 

Failure  Reporting  and  Corrective  Action  System 
Endless  Burn-In  Concept 
Reliability  Document  Impact  on  RADC 
System  Development  Phares 

Failure  Rate  Versus  Cumulative  Operating  Hours  for 
Duane's  Original  Data 

Duane  Plot  for  Reliability  Growth  of  an  Airborne 
Radar 

Duane  P  t  Showing  the  Initial  "Hook"  During  the 
Early  Time  Period 

Initial  Hook  in  Bathtub  Cu*ve  Showing  an  Initially 
low  Failure  Rate  (High  MTBF) 

Linear/Staircase  Plot  of  ROGT  Test  Data 

Options  Available  to  a  Program  Mai  lger  for  a  Fixed 
Reliability  Test  Time 

Reliability  Tests  as  a  Function  of  Contract  Type 

Planned  Reliability  Growth  (Continuous) 

Planned  Reliability  G.  owth  (Phase-^y-Phase) 

Reliability  Growth  Process  Showing  a  Decrease  in 
Reliability  ("DIPS")  at  Certain  Program  Milestones 

Different  Ways  of  Reaching  the  S^.ie  MTBF  Goal 

Plotted  Data  for  Test  Time  Calculation  Example 

Exponential  Law  for  the  Appearance  of  Systematic 
Failures 


V 


PAGE 

• 

7 

9 

10 

15 

.  •' 

16 

.  _  v 

47 

■ --i.il " 

49 

51 

i  V 

51 

-N 

:  - 

'•V.  . 

'.V'.'l 

52 

,  -  'v  i 

70 

•  ’  v  *  *.  **  » 

.  .  ** 

71 

73 

73 

‘o'-.-V;. 

76 

V  . 

86 

-■ 

90 

.■  .  -  v\ 

*i 

94 

- 1 


Percent  Increase  in  Acquisition  Cost -vs-Normali zed  MTBF  102 


FIGURE  TITLE  PAGE 

6.10  Reliability  Task  Cost  Relationships  103 


8.1  Comparison  of  Cumulative  Life  Cycle  Cost;  With  115 

and  Without  Specified  Reliability  Growth 
Test  Requirements 


1.0  Objective:  The  use  of  reliability  growth  testing  and  test-ana lyze- 
and-f'x  (TAAiF)  testing  has  become  widespread  within  the  Department  of 
Defense  as  a  complement  to  and  substitute  for  formal  reliability  qualifi¬ 
cation  testing.  Many  different  models,  tools  and  techniques  for  their  use 
have  been  presented  in  the  literature,  military  standards  and  handbooks. 
Still,  many  reliability  experts  within  DoD  question  the  utility  and  cost 
effectiveness  of  reliability  growth  testing  and  describe  it  as  rewarding 
contractors  for  sloppy  initial  designs.  The  objective  of  this  study  was  to 
tally  investigate  the  subject  of  reliability  growth  testing  to  enable  a 
better  understanding  by  reliability  engineers  as  well  as  to  present  guid¬ 
ance  for  its  potential  application  in  the  development  of  Air  Force  systems. 

2.0  Approach :  The  approach  used  in  performing  the  in-house  study 
included  the  following: 

A.  Existing  Department  of  Defense  and  Air  Force  regulations,  direc¬ 
tives,  standards,  handbooks  and  policies  were  reviewed  to  determine  their 
impact  on  the  forms  of  reliability  testing  under  study. 

B.  A  literature  search  regarding  *"611  ability  growth  testing  and 
test-analyze-and-f ix  testing  was  performed  to  determine  how  requirements 
have  been/are  being  implemented,  what  management  and  analysis  techniques 
have  been  developed  and  what  the  results  have  been  of  the  application  of 
those  techniques. 
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C.  Various  reliability  experts  (government/industry)  were  consul  :ed 
to  benefit  from  their  experience  in  applying  reliability  growth  testing. 
Opinions  and  data  were  sought  with  respect  to  applying  reliability  growth 
and  TAAF  testing. 


D.  DoD  research  and  development  data  bases  were  searched  to  deter¬ 
mine  what  R&D  study  efforts  are  currently  under  way  regarding  these  forms 
of  reliability  testing. 


E.  The  results  of  the  above  four  tasks  were  reviewed  and  analyzed  by 
an  objective  RADC  team  of  experienced  reliability  engineers  and  conclu¬ 
sions  were  developed. 


2.1  Issues:  While  reliability  growth  testing  is  being  applied  widely  in 
DoD  systems  development,  there  are  a  number  of  questions  that  are  often 
expressed  by  those  skeptical  of  its  effectiveness  which  can  be  summarized 
as  follows: 


Who  pays  for  the  reliability  growth  testing  (RDGT)?  Does  the 
government  end  up  paying  mors? 


Does  RDGT  allow  DoD  contractors  to  "get  away  with"  a  sloppy  init¬ 
ial  design  because  they  can  fix  it  later  at  the  government's 
expense? 


Should  reliability  growth  testing  be  dedicated  or  integrated? 


When  should  a  reliability  growth  test  begin? 

Should  reliability  growth  be  planned  for  beyond  the  FSED  phase? 

Should  the  equipment  operate  at  the  fully  specified  performance 
level  prior  to  the  start  of  ROGT? 

Should  all  development  programs  have  some  sort  of  reliability 
growth  testing? 

How  does  the  applicability  of  reliability  growth  testing  vary 
with  the  following  points  of  a  development  program? 

a.  Complexity  of  equipment  and  its  challenge  to  the  state-of- 
the-art. 

b.  Operational  environment 

c.  Quantity  of  equipment  to  be  produced 

What  growth  model (s)  should  be  used? 

What  starting  points  and  growth  rates  should  be  used  for 
planning? 
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How  much  tesv  time  (and  calendar  time)  will  be  required  to  conduct 

the  testing? 

Wjer  will  corrective  actions  be  implemented? 

How  will  failures  be  counted? 

Will  there  be  an  accept/reject  criteria? 

Should  the  contractor  be  responsible  for  intermediate  milestones? 

Can/should  growth  testing  be  incentivized? 

Does  the  type  of  contract  affect  RDGT  decisions? 


Lv.v. 


What  is  adequate  time  for  verifying  a  design  fix? 

What  is  the  relationship  between  an  RQT  and  RDGT? 

Who  will  do  the  growth  tracking?  How  and  to  whom  will  the 
results/status  be  reported? 

How  much  validity/confidence  should  be  placed  on  the  numerical 
results  of  RDGT? 
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Based  on  the  research  conducted,  an  a  .tempt  will  be  made  to  answer  many  of 
these  questions  In  the  remainder  of  the  report.  The  results  of  the  study 
are  organized  as  follows  in  the  remainder  of  the  report. 


3.0  Reliability  Growth  Testing  Terminology 
4.0  DoD  Policy  on  Reliability  Growth  Testing 
5.0  Reliability  Growth  Analysis 

6.0  Reliability  Growth  Management  Techniques 

7.0  Reliability  Growth  Application  Experience 

8.0  Conclusions 

3.0  Reliability  Growth  Testing  Terminology 

3.1  Reliability  Testing;  The  use  and  misuse  of  many  reliability  testing 
terms  necessitates  inclusion  of  the  Table  3-1  definitions.  It  should  be 
noted  that  Reliability  Growth  Testing  (RGT)  and  Reliability  Develop¬ 
ment/Growth  Testing  (RDGT)  are  used  synonymously  in  this  report.  Test- 
Analyze-and-Fix  (TAAF)  is  the  process  by  which  reliability  growth  is 
achieved  and,  in  itself,  does  not  necessarily  include  the  structured 
planning  and  tracking  associated  with  an  RG1 .  MIL-STD-785B  considers  the 
Reliability  Development/Growth  Test  as  an  engineering  test  while  the  other 
two  forms  of  reliability  testing  are  considered  accounting  tests.  Before 
considering  the  applicability  of  reliability  growth  testing,  some  prelimi¬ 
nary  concepts  need  to  be  addressed: 


t; 


TABLE  3-1:  MIL-STD-785B  RELIABILITY  TEST  DEFINITIONS 


Environmental  Stress  Screening  (ESS):  A  series  of  tests  conducted 
under  environmental  stresses  to  disclose  weak  parts  and  workman¬ 
ship  defects  for  correction. 

Reliability  Development/Growth  Test  (RD6T):  A  series  of  tests 
conducted  to  disclose  deficiencies  and  to  verify  that  corrective 
actions  will  prevent  recurrence  in  the  operational  inventory. 
(Also  known  as  "TAAF"  testing) 

Reliability  Qualification  Test  (RQT):  A  test  conducted  under  spe¬ 
cified  conditions,  by,  or  on  behalf  of,  the  government,  using 
items  representative  of  the  approved  production  configuration,  to 
determine  compliance  with  specified  reliability  requirements  as  a 
basis  for  production  approval.  (Also  known  as  a  "Reliability 
Demonstration,"  or  "Design  Approval"  test.) 

Production  Reliability  Acceptance  Test  (PRAT):  A  test  conducted 
under  specified  conditions,  by,  or  on  behalf  of,  the  government, 
using  delivered  or  deliverable  production  items,  to  determine  the 
producer's  compliance  with  specified  reliability  requirements. 


3.2  Growth  and  Failures:  PH  Mead  (Ref  5)  states  that  there  are  three 
distinct  ways  in  which  reliability  can  grow: 


"Growth  Mode  1.  By  operating  each  equipment  (or  portion  of  it)  to 
expose  and  eliminate  rogue  components  or  manufacturing  errors. 


Growth  Mode  2.  By  familiarization,  increased  operator  skill  and 
general  "settling  down"  in  manufacturing,  use  and  servicing. 


Growth  Mode  3.  By  discovering  and  correcting  errors  or  weaknesses  in 
design-  manufacturing  or  related  procedures." 
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Reliability  of  electronic  equipment  can  improve  both  at  the  collective  and 
individual  equipment  level.  Burn-in  improves  the  reliability  of  the 
equipment  subjected  to  it  while  design  changes  improve  (or  degrade)  the 
reliability  of  all  equipment  subject  to  the  changes.  Each  of  the  three 
growth  or  evolution  modes  can  be  made  more  effective  by  planned  activities. 

Regardless  of  how  well  the  reliability  of  an  equipment  is  designed  in,  the 
complexity  of  today's  electronics  make  it  impossible  to  foresee  all  errors 
and  imperfections.  Green  (Ref  3)  found  that  75%  of  all  systematic  design 
problems  could  not  be  foreseen  prior  to  testing.  Defects  or  failure  causes 
in  electronic  equipment  can  be  categorized  as  shown  in  Figure  3.1 


FIGURE  3.1:  CATEGORIZATION  OF  DEFECTS 
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Mead  defined  the  three  failure  classes  as: 

A.  Systematic  -  repetitive  (or  from  their  nature  liable  to  be 
repetitive). 

B.  Induced  -  Due  to  accident  from  causes  internal  or  external  to  the 
equipment. 

C.  Residual  -  Neither  of  the  above. 

A  constant  review  of  defects  is  necessary  to  ensure  that  random  and  induced 
categorized  events  aren't  alibis  for  performing  no  corrective  action.  He 
found  that  an  exponential  law  applied  to  the  appearance  of  systematic 
failures  in  complex  airborne  equipment.  Most  authors  speak  of  reliability 
growth  testing  as  a  means  of  eliminating  these  systematic  failures. 

3.3  Failure  Reporting  and  Corrective  Action  System  (FRACAS):  A  well 
accepted  military  reliability  program  task  is  a  closed  loop  FRACA  system  as 
shown  in  Figure  3.2.  The  reliability  growth  test  can  be  thought  of  as  a 
better  controlled  and  more  structured  form  of  a  FRACAS  system. 


3.4  Reliability  Growth  Limiting  Values:  Bezat  (Ref  6)  postulated  the 
sources  of  growth  to  be  two  categories,  (1)  reliability  growth  due  to 
conscious  corrective  action,  and,  (?)  "endless  burn-in"  maturing  factor. 
He  showed  that  growth  continues  to  a  limiting  reliability  level  even  with¬ 
out  further  design  corrective  activity.  The  idea  of  "endless  burn-in" 
means  that  "infant  mortality"  is  a  misnomer  and  that  the  magnitude  of  its 
effects  extend  far  out  in  life.  The  effect  was  categorized  as  follows: 

"Endless  Burn-In  includes  all  the  intangible  maturity  factors  associated 
with  undocumented  improvements  in  test,  repair,  build  processes,  and  con¬ 
trol  of  environment/application  to  original  objectives/"  Bezat  states 
that  the  instantaneous  failure  rate  of  an  LRU  includes  a  residual  component 
which  becomes  significant  only  when  the  average  age  of  the  LRU's  becomes 
about  2500  hours  (Fig  3.3). 


FIGURE  3.3:  ENDLESS  BURN-IN  CONCEPT 


100  1,000  10,000 
MEAN  AGE,  LRU  (OPERATING  HOURS) 
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3.5  Reliability  Growth  In  Management:  If  the  premise  of  reliability 
Improvement  through  design  change  is  believed,  the  question  becomes  how 
effective  Is  the  process  and  how  much  resources  are  required  to  meet  the 
reliability  requirements?  Meade  (Ref  8)  said:  "Reliability  growth  man¬ 
agement  facilitates  early  warning  by  helping  a  manager  in  at  least  four 
ways:  First  Is  the  preparation  of  planned,  time  phased  profiles  of  relia¬ 
bility  growth.  Next,  the  methodology  can  be  used  to  assess  reliability 
progress  against  „h1s  plan.  Third,  projections  of  reliability  trends  can 
be  developed.  Finally,  the  methodology  can  be  used  as  a  powerful  planning 
tool  for  determining  the  time  and  resources  needed  for  the  test  phases  of  a 
reliability  program  and  In  evaluating  the  impact  of  limitations  and 
changes  in  the  program."  In  the  context  of  reliability  growth  in  this 
report,  it  is  important  to  emphasize  that  growth  results  from  redesign 
effort  that  eliminates  failure  sources  that  were  discovered  through  analy¬ 
sis  of  test  results.  An  important  aiscinction  to  be  made  is  that  in  the 
burn-in  of  an  item,  defect ‘ >e  parts  are  replaced  with  good  parts  of  the 
same  design  resulting  in  an  improvea  reliability  of  the  one  unit  being 
burned-in.  Redesign  to  eliminate  failure  sources  involves  changing  the 
design  configuration  of  c.ll  units,  not  just  the  one  under  test. 

3.6  Reliability  Growth  vs  Other  Reliability  Tasks:  Mead  (Ref  5) 
described  as  a  necessity  for  a  successful  growth  process  "starting  with  a 
healthy  plant"  which  results  from  the  other  reliability  program  tasks.  The 
reliability  growth  management  process  provides  an  orderly  way  to  control 
the  development  process,  surface  problems  and  redirect  assets. 


3.7  No-Growth  Growth:  Clark  (Ref  42)  cautioned  against  the  misuse  of 
reliability  growth  concepts  by  Indicating  case  histories  which  had  been 
previously  portrayed  as  reliability  growth  in  the  literature  that  really 
weren't.  In  his  work  he  referred  to  situations  where  growth  was  portrayed 
by  using  reliability  demonstration  data  and  individual  equipment  burn-in 
data  as  "no-growth  growth."  These  were  misapplications  of  growth  manage¬ 
ment  and  he  cautioned,  "to  effect  a  growth  in  inherent  reliability,  one  or 
more  of  tne  basic  design  or  process  parameters  (number  and  types  of  compon¬ 
ent  parts,  their  material  quality  and  stress  levels  and  structural  and 
thermal  characteristics)  must  be  improved."  An  example  of  no-growth 
growth  would  be  the  purging  of  systematic  failures  from  reliability  demon¬ 
stration  test  data  to  show  what  the  system  reliability  could  be  if  a 
perfect  fix  could  be  found  for  these  problems.  Unless  the  fixes  are 
actually  implemented  and  proven,  you  will  have  a  case  of  no-growth  growth. 


3.8  Reliability  Growth  Misconceptions:  In  order  to  further  clarify  reli¬ 
ability  growth  it  is  important  to  point  out  the  following  misconceptions 
regarding  it: 


A.  Reliability  growth  is  a  naturally  occurring  phenomenon  in  elec¬ 
tronic  equipment.  (It  is  not) 


B.  Reliability  growth  occurs  as  a  natural  course  of  events  after  a 
system  is  introduced  into  the  operational  inventory.  (It  does  not) 


C.  Equipment  burn-in  to  remove  Infant  mortality  type  failures  causes 
reliability  growth.  {It  does  not,  except  for  that  particular  equipment) 


D.  Replacing  early  equipment  failures  with  good  parts  to  repair  the 
observed  weaknesses  causes  reliability  growth.  (It  does  not) 

E.  Reliability  predictions  that  improve  with  mere  detailed  design 
disclosure  reflect  reliability  growth.  (They  do  not) 

In  the  context  of  this  report,  reliability  growth  is  the  result  of  the 
iterative  process  of  sample  testing;  identification  of  design,  part  and 
workmanship  defects;  and  correction  of  the  causes  of  these  defects.  The 
basic  equipment  design  establishes  the  point  from  which  reliability  growth 
starts  and  the  upper  bound  on  potential  reliability. 

4.0  DoD  Policy  on  Reliability  Growth  Testing 

4.1  Standards:  Reliability  as  an  engineering  discipline  is  controlled  by 
a  series  of  directives,  regulations,  standards,  handbooks  and  policies 
within  the  DoD  acquisition  and  development  arena.  Some  of  these  are 
triservice  (apply  to  all  DoD  components)  others  are  uniquely  designeu  for 
one  or  more  services'  use.  Table  4-1  is  a  representation  of  these  docu¬ 
ments.  Figure  4.1  shows  a  hierarchy  of  how  RADC,  in  particular,  is 
effected  by  these  reliability  documents  on  development  and  acquisition 
programs. 
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TABLE,  l-l :  DOD  RELIABILITY  RELATED  DOCUMENTS  (RELIABILITY  TEST  IMPACT) 


NUMBER 


TITLE 


ML-STD-785B 


DoD  5000.40  Reliability  and  Maintainability  (8  July  1980) 

AIR  800-18  Air  Force  Reliability  and  Maintainability  Pro¬ 

gram  (15  June  1982) 

ML-STD-785B  Reliability  Program  for  Systems  and  Equipment 
Development  and  Production  (15  September  1980) 

MIL-STD-781C  Reliability  Design,  Qualification  and  Produc¬ 
tion  Acceptance  Tests:  Exponential  Distribu¬ 
tion  (21  October  1977) 

r  *  I L - ST  D - 7  2 1 C  Definitions  of  Terms  for  Reliability  and  Main¬ 

tainability  (12  June  1982) 

MIL-STD- 1635  Reliability  Growth  Testing  (3  February  1978) 


'in  r*T  r\  nnrn 
I'llL-OI  U-4UDO 


rceiiaDinty  Development  lests  (21  March  19/7) 


MIL-HDBK-189  Reliability  Growth  Management  (13  February 


*.  A.  *\, 


FIGURE.  4.1:  RELIABILITY 


PROCESS 


IMPACT  or*  RADC 


4.2  Development  Process:  In  the  context  of  discussions  regarding  acqui¬ 
sition  and  development  programs  within  the  Air  Force,  confusion  sometimes 
exists  with  regard  to  the  pregram  development  phases.  Figure  4.2  clarifies 
how  these  phases  are  Interrelated.  It  Is  on  the  basis  of  where  a  particu¬ 
lar  program  Is  in  relation  to  a  potential  production  decision  that  deter¬ 
mines  the  tailoring  of  reliability  program  tasks.  Programs  have  been  known 
to  go  directly  from  an  Advanced  Development  Model  to  Production.  For  this 
reason  RADC  has  structured  Its  reliability  task  tailoring  guidance  In 
terms  of  the  following: 

FIGURE  4.2:  SYSTEM  DEVELOPMENT  PHASES 


PDR 


♦Defense  Systems  Acquisition  Review  Council 


4.2.1  Reliability  Development  Phases: 

A.  Pre-Reli ability  Phase:  Those  early  phases  in  a  development  pro¬ 
cess  where  no  structured  reliabilty  tasks  are  appropriate. 

B.  Reliability  Study  Phase •,  This  early  phase  has  reliability  acti¬ 
vities  related  to  trade  studies  accessing  the  reliability  potential  of 
various  system  configurations. 

C.  Reliability  Design/Analysis  Phase:  This  phase  begins  the  sig¬ 
nificant  application  of  reliability  engineering  tasks  to  the  system  devel¬ 
opment.  Activities  will  provide  the  framework  for  the  next  phase  < usually 
FSED).  It  is  riot  the  last  development  phase  Defore  a  potential  production 
decision. 

D.  Reliability  Definition  and  Demonstration  Phase:  This  phase  is 
the  final  development  process  prior  to  a  production  decision.  Reliability 
engineering  is  a  major  part  of  this  phase's  development  process.  Reliabi¬ 
lity  quantitative  parameters  are  specified,  predicted  and  demonstrated. 

E.  Reliability  Assurance  Phase:  This  phase  is  the  build,  test  and 
deliver  of  the  reliability  designed  in  during  prior  development.  Reliabi¬ 
lity  activities  are  devoted  mainly  to  "assurance"  type  tasks  such  as  envi¬ 
ronmental  stress  screening  and  production  reliability  acceptance  testing. 


Table  4-2  has  beeh  extracted  from  MIL-ST0-785B  "Reliability  Program  For 
Systems  and  Equipment  Development  and  Production"  to  show  how  particular 
reliability  tasks  are  to  be  tailored  for  a  particular  development  phase. 

The  terminology  used  for  phase  definitions  of  Table  4-2  are  that  of  AFR 
800-1  "Major  System  Acquisitions."  Many  RADC  development  programs  are 
covered  by  the  AFR  "80"  series  regulations  with  such  phases  as  "exploratory 
development,"  "advanced  development,"  "engineering  development"  and 
others.  In  some  instances  phases  are  omitted  from  the  development  cycle. 
A  program  can  transition  directly  from  an  advanced  development  model  (ADM) 
to  production.  Therefore,  the  key  to  effective  implementation  of  reliabi¬ 
lity  requirements  and  tasks  is  not  in  tying  them  to  development  phase  names 
but  in  defining  them  in  terms  of  how  close  the  development  phase  is  to  a 
production  decision  which  must  include  reliability  consideration.  Table 
4-3  indicates  the  general  reliability  considerations  as  a  function  of 
reliability  design  phase  terminology. 

4.3  Tailoring  Tasks?  While  MIL-STD-78FB  recommends  reliability  tasks  for 
the  various  phases  of  development,  as  indicated  by  Table  4-2,  it  is  impor¬ 
tant  to  note  ttiat  each  program  is  different  in  terms  of  funding/schedule, 
equipment  performance  requirements,  challenge  tc  the  state-of-the-art,  and 
personnel  and  contractors  involved.  Therefore,  a  "boiler  plate"  approach 
to  reliability  is  never  the  correct  approach.  Recently,  RADCs  reliabi¬ 
lity  experts  prioritized  standard  reliability  tasks  in  accordance  with 
their  payoff  for  varying  environments  and  development  phases.  Table  4-4 
shows  the  results.  These  results  were  based  on  a  mix  of  the  "80"  series 


and  "800"  series  AF  regulations  terminology  in  that  the  phases  ADM-FSED- 
PROD  are  considered.  After  recognizing  (as  previously  pointed  out)  that 
there  are  cases  where  an  ADM  goes  directly  to  production  without  further 
development,  RADC  formulated  reliability  task  application  guidelines  based 
on  the  reliability  phase  terminology.  These  results  are  represented  by 
Table  4-5.  In  line  with  all  recent  reliability  literature,  the  emphasis  is 
placed  on  "up  front"  reliability  engineering  tasks,  rather  than  reliabi¬ 
lity  accounting  tasks. 

4.4  Direction :  While  tailoring  is  key  to  successful  cost  effective  reli¬ 
ability  accomplishment,  certain  reliability  aspects  are  required  by  relia¬ 
bility  directives,  regulations  and  standards.  The  following  paragraphs 
address  how  the  documents  of  Table  4-1  relate  to  reliability  growth  and 
TAAF  testing. 
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TABLE  4-2:  APPLICATION  MATRIX  FOR  PROGRAM  PHASES 


V 


s 

A".' 


tas;: 

TITLE 

TASK 

TYPE 

PROGRAM  PHASE  j 

CONCEPT 

VALID 

FSED 

PROD 

101 

RELIABILITY  PROGRAM  PLAN 

MGT 

S 

S 

G 

G 

102 

MONITOR/CONTROL  OF  SUBCONTRACTORS 

AND  SUPPLIERS 

MGT 

S 

S 

R 

G 

103 

PROGRAM  REVIEWS 

MGT 

S 

S(2) 

G(2) 

G(2) 

104 

FAILURE  REPORTING,  ANALYSIS.  AND 
CORRECTIVE  ACTION  SYSTEM  (FRACAS) 

ENG 

NA 

S 

G 

G 

105 

FAILURE  REVIEW  BOARD  (FRB) 

MGT 

NA 

S(2) 

G 

G 

201 

RELIABILITY  MODELING 

ENG 

S 

S(2) 

G(2) 

GC  ( 2 ) 

202 

RELIABILITY  ALLOCATIONS 

ACC 

S 

G 

G 

GC 

203 

RELIABILITY  PREDICTIONS 

ACC 

s 

S(2) 

G(2) 

GC(2) 

204 

FAILURE  MOOES,  EFFECTS,  AND 
CRITICALITY  ANALYSIS  (FMECA) 

ENG 

s 

(1)(2) 

G 

OH?) 

GC 

(1 )  (2) 

205 

SNEAK  CIRCUIT  ANALYSIS  (SCA) 

ENG 

NA 

NA 

GO) 

GCO) 

206 

ELECTRONICS  PARTS/CIRCUITS 

TOLERANCE  ANALYSIS 

ENG 

NA 

NA 

G 

GC 

207 

PARTS  PROGRAM 

ENG 

S 

(2)0) 

G 

(2) 

G 

(2) 

208 

RELIABILITY  CRITICAL  ITEMS 

MGT 

S(l) 

S(l) 

G 

C 

209 

EFFECTS  OF  FUNCTIONAL  TESTING, 
STORAGE,  HANDLING,  PACKAGING, 
TRANSPORTATION,  AND  MAINTENANCE 

ENG 

NA 

S(1) 

G 

GC 

301 

ENVIRONMENTAL  STRESS  SCREENING  (ESS) 

ENG 

NA 

S 

G 

G 

302 

RELIABILITY  DEVELOPMENT /GROWTH 
TESTING 

ENG 

NA 

3(2) 

G(2) 

NA 

303 

RELIABILITY  QUALIFICATION  TEST 
(RQT)  PROGRAM 

ACC 

NA 

S(2) 

G(2) 

G(2) 

304 

PRODUCTION  RELIABILITY  ACCEPTANCE 
ACCEPTANCE  TEST  (PRAT)  PROGRAM 

ACC 

r 

NA 

S 

(2)(3) 

CODE  DEFINITIONS 


TASK  TYPE:  PROGRAM  PHASE: 

ACC  -  RELIABILITY  ACCOUNTING  S  -  SELECTIVELY  APPLICABLE 

ENG  -  RELIABILITY  ENGINEERING  ‘  G  -  GENERALLY  APPLICABLE 

MGT  -  MANAGEMENT  GC  -  GENERALLY  APPLICABLE  TO  DESIGN 

CHANGES  ONLY 

NA  -  r.3T  APPLICABLE 

0)  -  REQUIRES  CONSIDERABLE  INTERPRETATION 
OF  INTENT  TO  BE  COST  EFFECTIVE 

(Z)  -  MIL-STD-785  IS  NOT  THE  PRIMARY 

IMPLEMENTATION  REQUIREMENT.  OTHER 
MIL-STDS  OR  STATEMENT  OF  WORK 
REQUIREMENTS  MUST  BE  INCLUDED  TO 
DEFINE  THE  REQUIREMENTS. 
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4.4.1  DoD  Directive  5000.40  "Reliability  and  Maintainability"  (8  Jul 
80) :  This  directive  requires  a  "balanced  mix"  of  reliability  engineering 
and  accounting  tasks  tailored  for  maximum  efficiency.  Under  the  reliabi¬ 
lity  engineering  policy,  reliability  growth  testing  is  listed  as  a  design 
fundamental  to  "disclose  design  deficiencies  and  to  verify  the  effective¬ 
ness  of  corrective  action0  "  The  di  active  further  states  that  "require¬ 
ments  and  achievements  for  each  applicable  system  R&M  parameter  shall  be 
numerically  traceable:  (a)  through  all  phases  of  the  system  life  cycle, 
..."  It  emphasizes  the  importance  of  reliability  growth  as  a  high  payoff 
reliability  engineering  task  by  stating: 

"R&M  growth  is  required  during  full  scale  development,  concurrent  de¬ 
velopment  and  production  (where  concurrency  is  approved),  and  during 
initial  deployment.  Predicted  R&M  growth  shall  be  stated  as  a  series 
of  intermediate  milestones,  with  associated  goals  and  thresholds,  for 
each  of  these  phases." 

"A.  A  period  of  testing  shall  be  scheduled  in  conjunction  with 
each  intermediate  milestone  The  purpose  of  these  tests  shall  be 
to  find  design  deficiencies  a"d  manufacturing  defects.  A  block 
of  time  and  resources  shall  be  scheduled  for  the  correction  of 
deficiencies  and  defects  found  by  each  period  of  testing,  to 
prevent  their  recurrence  in  the  operational  inventory.  Adminis¬ 
trative  delay  of  R&M  engineering  change  proposals  shall  be 
minimized. 
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B.  The  differences  between  required  values  for  system  R&M  para¬ 
meters  shall  be  used  to  concentrate  R&M  engineering  effort  where 
it  is  needed  (for  example,  enhance  mission  reliability  by  cor¬ 
recting  mission-critical  failures;  reduce  maintenance  manpower 

st  by  correcting  any  failures  that  occur  frequently). 

C.  Approved  R&M  growth  shall  be  assessed  and  enforced.  Enfor¬ 
cement  of  intermediate  R&M  goals  ..hall  be  left  to  the  acquiring 
activity.  Failure  to  achieve  an  intermediate  R&M  threshold  is  a 
projected  threshold  breach,  and  if  it  occurs,  an  immediate  review 
by  the  program  decision  authority  is  required." 

With  regard  to  reliability  demonstration,  the  directive  says  "R&M  demon¬ 
stration,  qualification  tests  and  acceptance  tests  shall  be  tailored  for 
effectiveness  and  efficiency  (maximum  return  on  cost  and  schedule  invest¬ 
ment)  in  terms  of  management  information  they  provide."  Reliability 
growth  testing  is  considered  an  engineering  task  while  reliability  demon¬ 
stration  testing  is  considered  an  accounting  task.  Accounting  tasks 
measure  reliability  (demonstrate  a  value)  while  engineering  tasks  improve 
reliability. 

4.4.2  AFR  800-18:  "Air  Force  Reliability  and  Maintainability  Program 
(15  June  1982):  This  document  is  intended  to  revise  the  previous  AF 
Regulation  80-5  to  comply  with  DoD  5000.40.  Requirements  of  DoD  5000.40 
are  restated  with  phrases  such  as  "...it  is  necessary  to  address  R&M 
thresholds  at  each  program  decision  milestone.  These  thresholds  will  be 
derived  from  mature  system  requirements,"  and  "each  R&M  program  will 


include  a  balanced  mix  of  R&M  engineering  and  accounting  tasks.  Early 
investment  shall  be  made  in  R&M  engineering.  R&M  accounting  will  provide 
management  information.  Cost  and  schedule  investment  in  the  R&M  program 
will  be  clearly  visible  and  carefully  controlled."  Reliability  growth  is 
implied  by  such  statements  as  "terms  are  expressed  in  mature  system  values 
along  with  interim  thresholds." 

The  regulation  states  for  Full  Scale  Development  (Full  Scale  Engineering 
Development)  (from  Milestone  II  to  Production  Decision)  "a  numerical  value 
for  each  selected  (reliability  requirement)  is  determined,  contractual ly 
specified,  and  verified  by  test  prior  to  a  production  decision.  Testing 
will  be  scheduled  to  allow  enough  time  to  review  the  results  prior  to  the 
production  decision."  It  further  states: 

"For  each  R&M  characteristic  identified  at  Milestone  II,  projected 
reliability  growth  curves  are  established  and  used  by  the  program 
manager  to  manage  the  growth  process.  The  purpose  of  the  yrowth 
program  will  be  to  insure  that  testing  is  programmed  to  find  design 
deficiencies  and  manufacturing  defects,  that  time  and  resources  are 
scheduled  to  correct  deficiencies  and  defects,  and  that  corrective 
design  changes  are  implemented  and  verified." 

A..  Projected  growth  must  show  achievement  of  the  threshold  values  of 
R&M  characteristics  at  intermediate  milestones  and  at  the  completion  of 
full  scale  development  testing  so  the  achieved  values  can  be  reviewed  at  a 
production  decision  point. 
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B.  Growth  curves  shall  not  be  used  to  predict  achievement  of 
requirements  in  the  production  phase  unless  either  concurrent  development 
and  production  are  specifically  authorized,  or  funds  have  been  identified 
to  correct  specific  R&M  deficiencies. 

C.  A  projected  growth  curve  is  established  for  each  contractually 
specified  parameter.  These  curves  must  show  adequate  progress  to  achieve 
the  specified  value  before  commencement  of  reliability  qualification 
testing. 

D.  Use  test-analyze-and-fix  (TAAF )  techniques  to  accomplish  neces¬ 
sary  reliability  growth.  Actual  growth  will  be  tracked  through  monitoring 
of  functional,  environmental,  and  evaluation  testing  conducted  during 
development.  However,  specific  rel iabi lity  growth  tests,  such  as  Combined 
Environmental  Reliability  Test  (CERT),  should  be  conducted  when  compatible 
with  the  ovetill  program  schedule."  (This  applies  also  for  concurrent  FSD 
and  production). 

The  regulation  defines  the  FSD  program  by: 

"The  FSD  program  is  intended  to  mature  the  system  R&M  characteristics 
as  soon  as  possible  by  finding  and  correcting  design  deficiencies, 
reducing  producibi 1 i ty  risks  and  by  identifying  and  pursuing  R&M 
improvement  opportunities.  To  do  this: 
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A.  The  approved  design  approach  shall  be  matured  through  devel¬ 
opment  testing  of  equipment  and  the  incorporation  of  specific 
design  Improvements. 

B.  Ihe  maturation  process  shall  be  monitored  through  growth 
tracking  and  design  review  ev  luations." 

4.4.3  MIL-STD-785B  'Reliability  Programs  for  Systems  and  Equipment 
Development  and  Production11  (15  Sep  80):  This  revision  of  the  main  DoD 
reliability  standard  oresents  a  "shopping  list"  of  reliability  tasks  to  be 
tailored  to  a  given  application.  The  recommendations  given  for  task  appli¬ 
cation  were  already  cited  in  Table  4-2.  Increased  emphasis  (over  MIL-STD- 
785A)  is  placed  on  reliability  engineering  tasks  and  tests  with  the  thrust 
toward  prevention,  detection,  and  correction  of  design  deficiencies,  weak 
parts  and  workmanship  defects.  This  standard  stresses  reliability 
engineering: 

"Reliability  Engineering.  Tasks  shall  focus  ori  the  prevention, 
detection,  and  correction  of  reliability  design  deficiencies,  weak 
parts,  and  workmanship  defects.  Reliability  engineering  shall  be  an 
integral  part  of  the  item  design  process,  including  design  changes. 
The  means  by  which  reliability  engineering  contributes  to  the  design, 
and  the  level  of  authority  and  constraints  on  this  engineering  dis¬ 
cipline,  shall  be  identified  in  the  reliability  program  plan.  An 
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efficient  reliability  program  shall  stress  early  investment  in  relia¬ 
bility  engineering  tasks  to  avoid  subsequent  costs  and  schedule 
delays. " 

With  respect  to  demonstration  of  contractual  reliability  requirements 
(electronics),  the  standard  states  "conformance  to  the  minimum  acceptable 
MTBF  requirement  shall  be  demonstrated  by  tests  selected  from  MI L-STD-78 1 , 
or  alternative  specified  by  the  PA  (procuring  activity)."  Reproduced  for 
completeness  as  Tables  4-6,  4-7  and  4-8  are  respectively:  Task  104, 
"Failure  Reporting,  Analysis,  and  Corrective  Action  System";  Task  302, 
"Reliability  Development/Growth  Test  (RDGT)  Program";  Task  303,  "Reliabi¬ 
lity  Qualification  Test  (RQT)  Program." 


TABLE  4-6: 


TASK  104  -  FAILURE  REPORTING,  ANALYSIS  AND  CORRECTIVE 
ACTION  SYSTEM  (FRACAS) 


104.1  Purpose.  The  purpose  of  task  104  Is  to  establish  a  closed  loop  failure  reporting 
system,  procedures  for  analysis  of  failures  to  determine  cause,  and  documentation  for  record¬ 
ing  corrective  action  taken. 

104.2  Task  Description 

104.2.1  The  contractor  shall  have  a  closed  loop  system  that  collects,  analyzes,  and  records 
failures  that  occur  for  specified  levels  of  assembly  prior  to  acceptance  of  the  hardware  by 
the  procuring  activity.  The  contractor's  existing  data  collection,  analysis  and  corrective 
action  system  shall  be  utilized,  with  modification  only  as  necessary  to  meet  the  requirements 
specified  by  the  PA. 

104.2.?  Procedures  for  Initiating  failure  reports,  the  analysis  of  failures,  feedback  of 
corrective  action  Into  the  design,  manufacturing  and  test  processes  shall  be  Identified. 
Flow  dlagram(s)  depicting  failed  hardware  and  data  flow  shall  also  be  documented.  The 
analysis  of  failures  shall  establish  and  categorize  the  cause  of  failure. 

104.2.3  The  closed  loop  system  shall  Include  provisions  to  assure  that  effective  corrective 
actions  are  taken  on  a  tlmfely  basis  by  a  follow-up  audit  that  reviews  all  open  failure 
reports,  failure  analyses,  and  corrective  action  suspense  dates,  and  the  reporting  of  delin¬ 
quencies  to  management.  The  failure  cause  for  each  failure  shall  be  clearly  stated. 

104.2.4  When  applicable,  the  method  of  establishing  and  recording  operating  time,  or  cycles, 
on  equipments  shall  be  clearly  defined. 

104.2.5  The  contractor's  closed  loop  failure  reporting  system  data  shall  be  transcribed  to 
Government  forms  only  If  specifically  required  by  the  procuring  activity. 

104.3  Details  to  be  Specified  by  the  PA  (reference  1.2. 2.1) 

104.3.1  Details  to  be  specified  in  the  SOW  shall  include  the  following,  as  applicable: 

a.  Identification  of  the  extent  to  which  the  contractor's  FRACAS  must  be  compa¬ 
tible  with  PA's  data  system. 

(R)  b.  Identification  of  level  of  assembly  for  failure  reporting. 

c.  Definitions  for  failure  cause  categories. 

d.  Identification  of  logistic  support  requirements  for  LSAR. 

e.  Delivery  identification  of  any  data  Item  required. 
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TABLE  4-7:  TASK  302  -  RELIABILITY  DEVELOPMENT /GROWTH  (RDGT)  PROGRAM 


302.1  Purpose .  The  purpose  of  task  302  Is  to  conduct  pre-qualification  testing  (also 
known  as  TAAF)  to  provide  a  basis  for  resolving  the  majority  of  reliability  problems  early  In 
the  development  phase,  and  Incorporating  corrective  action  to  preclude  recurrence,  prior  to 
the  start  of  production, 

302.2  Task  Description 

302.2.1  A  reliability  development/ growth  test  (TAAF  test)  shall  be  conoucted  for  the  purpose 
of  enhancing  system  reliability  through  the  Identification,  analysis,  and  correction  of 
failures  and  the  verification  of  the  corrective  action  effectiveness.  Mere  repair  of  the 
test  Item  does  not  constitute  corrective  action. 

302.2.1.1  To  enhance  mission  >"sl lability,  corrective  action  shall  be  focused  on  mission- 
critical  failure  modes.  To  enhance  basic  reliability,  corrective  action  shall  be  focused  on 
the  most  frequent  failure  modes  regardless  of  their  mission  criticality.  These  efforts  shall 
be  balanced  to  meet  predicted  growth  for  both  parameters. 

302.2.1.2  Growth  testing  will  emphasize  performance  monitoring,  failure  detection,  fail¬ 
ure  analysis,  and  the  incorporation  and  verification  of  design  corrections  to  prevent  recur¬ 
rence  of  failures. 

302.2.2  A  TAAF  test  plan  shall  be  prepared  and  shall  Include  the  following,  subject  to  PA 
approval  prior  to  Initiation  of  testing: 

a.  Test  objectives  and  requirements.  Including  the  selected  growth  model  and 
growth  rate  and  the  rationale  for  both  selections. 

b.  Identification  of  the  equipment  to  be  tested  and  the  number  of  test  items  of 
each  equipment. 

c.  Test  conditions,  environmental,  operational  and  performance  profiles,  and  the 
duty  cycle. 

d.  Test  schedules  expressed  in  calendar  time  and  Item  life  units,  including  the 
test  milestones  and  tesc  program  review  schedule. 

e.  Test  ground  rules,  chargeabi llty  criteria  and  Interface  boundaries. 

f.  Test  facility  and  equipment  descriptions  and  requirements. 

g.  Procedures  and  timing  for  corrective  actions. 

h.  Blocks  of  time  and  resources  designated  for  the  incorporation  of  design 
corrections. 

1.  Data  collection  and  recording  requirements. 

j.  FRACAS. 

k.  Government  furnished  property  requirements. 

l.  Description  of  preventive  maintenance  to  be  acconvllshed  during  test. 

m.  Final  disposition  of  test  Items. 

n.  Any  other  relevant  considerations. 

302.2.3  As  specified  by  the  procuring  activity,  the  TAAF  test  plan  shall  be  submitted  to  the 
procuring  act.vity  for  its  review  and  approval.  This  plan,  as  approved,  shall  be  Incorpor¬ 
ated  into  the  contract  and  shall  become  the  basis  for  contractual  compliance. 

302 . 3  Details  to  be  Specified  by  the  PA  (reference  1. 2.2.1) 

302.3.1  Details  to  be  specified  in  the  SOW  shall  Include  the  following,  as  applicable: 

(R)  a.  Imposition  of  task  104  as  a  requisite  task. 

(R)  b.  Identification  of  a  I Ife/mission/environmental  profile  to  represent  equipment 
usage  in  service. 

c.  Identification  of  equipment  and  quantity  to  be  used  for  reliability  devel¬ 
opment/growth  testing. 

d.  Delivery  identification  of  any  data  items  required.  __ __ — ^ M 
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TASK  4-8:  TASK  303  -  RELIABILITY  QUALIFICATION  TEST  (RQT)  PROGRAM 


303.1  Purpose.  The  purpose  of  task  303  Is  to  determine  that  the  specified  reliability 
ents  hav< 


requirements  have  been  achieved. 

303.2  Task  Description 

303.2.1  Reliability  qualification  tests  shall  be  conducted  on  equipments  which  shall  be 
Identified  by  the  PA  and  which  shall  be  representative  of  the  approved  production  config¬ 
uration.  The  relalbillty  qualification  testing  may  be  integrated  with  the  overall  sys¬ 
tem/equipment  qualification  testing,  when  practicable,  for  cost-effectiveness;  the  RQT  plan 
shall  so  Indicate  in  this  case.  The  PA  shall  retain  the  right  to  disapprove  the  test  failure 
relevancy  and  chargeabliity  determinations  for  the  reliability  demonstrations. 

303.2.2  An  RQT  plan  sha1!  be  prepared  in  accordance  with  the  requirements  of  MIL-STD-781,  or 
alternative  approved  by  the  PA,  and  shall  include  the  following,  subject  to  PA  approval  prior 
to  Initiation  of  testing: 

a.  Test  objectives  and  selection  rationale. 

b.  Identification  of  the  equipment  to  be  tested  (with  identification  of  the  com¬ 
puter  programs  to  be  used  for  the  test,  if  applicable)  and  the  number  of  test 
items  of  each  equipment. 

c.  Test  duration  and  the  appropriate  test  plan  and  test  environments.  The  test 
plan  and  test  environments  (if  life/mission  profiles  are  not  specified  by  the 
PA)  shall  be  derived  from  MIL-STD-781.  If  it  is  deemed  that  alternative 
procedures  are  more  appropriate,  prior  PA  approval  shall  be  requested  with 
sufficient  selection  rationale  to  permit  procuring  activity  evaluation. 

d.  A  test  schedule  that  is  reasonable  and  feasible,  permits  testing  of  equipment 
which  are  representative  of  the  approved  production  configuration,  and  allows 
sufficient  time,  as  specified  in  the  contract,  for  PA  review  and  approval  of 
each  test  procedure  and  test  setup. 

303.2.3  Detailed  test  procedures  shall  be  prepared  for  the  tests  that  are  included  in  the 
RQT  plan. 

303.2.4  As  specified  by  the  procuring  activity,  the  RQT  plan  and  test  procedures  shall  be 
submitted  to  the  procuring  activity  for  its  review  and  approval.  These  documents,  as 
approved,  shall  be  incorporated  into  the  contract  and  shall  become  the  basis  for  contractual 
compliance. 

303.3  Details  to  be  Specified  by  the  PA  (reference  1 .2.2. 1 ) 


303.3.1  Details  to  be  specified  in  the  SOW  shall  include  the  following,  as  applicable: 

(R)  a.  Identification  of  equipment  to  be  used  for  reliability  qualification  testing. 

(R)  b  Identification  of  MIL-STD-781,  MIL-STD-105  or  alternative  procedures  to  be 
used  for  conducting  the  RQT  (l.e.,  test  plan,  test  conditions,  etc.). 

c.  Identification  of  a  life/mission/environmental  profile  to  represent  equipment 
usage  in  service. 

d.  Logistic  support  coordinated  reporting  requirements  for  LSAR. 

e.  Delivery  identification  of  any  data  items  required. 
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The  standard  cites  three  objectives  of  a  reliability  test  program  as: 


A.  Disclose  deficiencies  in  item  design,  material  and  workmanship. 

B.  Provide  measured  reliability  data  as  input  for  estimates  of  oper¬ 
ational  readiness,  mission  success,  maintenance  manpower  cost  and  logis¬ 
tics  support  cost. 
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C.  Determine  compliance  with  quantitative  reliability  requirements. 

This  is  the  priority  order  of  the  objectives  to  be  met  subject  to  cost  and 
schedule  constraints.  The  previously  mentioned  tasks  (302  and  303)  along 
with  Task  301,  "Environmental  Stress  Screening"  and  Task  304,  "Production 
Reliability  Acceptance  Testing"  are  the  elements  of  a  reliability  test 
program  to  be  tailored  to  accomplish  the  above  objectives.  The  standard 
says  "a  properly  balanced  reliability  program  will  emphasize  ESS  and  RDGT, 
and  limit.,  hut.  not  eliminate,  RQT  and  PRAT." 


V 
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This  is  in  line  with  emphasis  on  engineering  tasks  and  "up  front"  reliabi¬ 
lity  spending.  Integrated  testing  is  stressed  with  environmental  tests 
(MIL-STD-810)  considered  as  the  early  portion  of  RDGT.  With  regard  to  the 
use  of  ESS  and  RDGT  as  methods  of  determining  contractual  compliance,  the 
standard  states:  "ESS  and  RDGT  must  not  include  accept/reject  criteria 
that  penalizes  the  contractor  in  proportion  to  the  number  of  failures  he 
finds,  because  this  would  be  contrary  to  the  purpose  of  the  testing  so 
these  tests  must  not  use  statistical  test  plans  that  establish  such 
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criteria.  RQT  and  PRAT  must,  provide  a  clearly  defined  basis  for  determin¬ 
ing  compliance,  but  they  must  also  be  tailored  for  effectiveness  and  effic¬ 
iency  {maximum  return  on  cost  and  schedule  investment)  in  terms  of  the 
management  information  they  provide." 


TABLE  4-9:  MIL-STD-785B  RELIABILITY  GROWTH  APPLICATION  GUIDANCE 


50.3.2.2  Reliability  development/qrowth  testing  (RDGT)  (task  302).  RDGT  Is  a  planned,  pre- 
qualiflcation,  test-analyze-and-fix  process.  In  which  equipment  are  tested  unde'  actual, 
simulated,  or  accelerated  environments  to  disclose  design  deficiencies  and  defects.  This 
testing  Is  Intended  to  provide  a  basis  for  early  incorporation  of  corrective  actions,  and 
verification  of  their  effectiveness,  thereby  promoting  reliability  growth.  However: 

TESTING  DOES  NOT  IMPROVE  RELIABILITY.  ONLY  CORRECTIVE  ACTIONS  THAT  PREVENT  THE  RECURRENCE  OF 
FAILURES  IN  THE  OPERATIONAL  INVENTORY  ACTUALLY  IMPROVE  RELIABILITY. 

50.3.2.2.1  It  Is  DoO  policy  that  reliability  growth  Is  required  during  full-scale  develop¬ 
ment,  concurrent  development  and  production  (where  concurrency  is  approved)  and  during  Init¬ 
ial  deployment.  Predicted  reliability  growth  shall  be  stated  as  a  series  of  Intermediate 
milestones,  with  associated  goals  and  thresho-lds,  for  each  of  those  phases.  A  period  of 
testing  shall  be  scheduled  in  conjunction  with  each  Intermediate  milestone.  A  block  of  time 
and  resources  shall  be  scheduled  for  the  correction  of  deficiencies  and  defects  found  by 
each  period  of  testing,  tc  prevent  their  recurrence  In  the  operational  Inventory.  Adminis¬ 
trative  delay  of  reliability  engineering  change  proposals  shall  be  minimized.  Approved 
reliability  growth  shall  be  assessed  and  enforced. 

50.3.2.2.2  Predicted  reliability  growth  must  differentiate  between  the  apparent  growth 
achieved  by  screening  weak  parts  and  workmanship  defects  out  of  the  test  Items,  and  the  step- 
function  growth  achieved  by  design  corrections.  The  apparent  growth  does  not  transfer  from 
prototypes  to  production  units;  instead,  it  repeats  In  every  Individual  item  of  equipment. 
The  step-function  growth  does  transfer  to  production  units  that  Incorporate  effective  design 
corrections.  Therefore,  ROGT  plans  should  Include  a  series  of  test  periods  (apparent 
growth),  and  each  of  the  test  periods  should  be  followed  by  a  "fix"  period  (step-function 
growth).  There  two  or  more  items  are  being  tested,  their  "test"  and  "fix"  periods  should  be 
out  of  phase,  so  one  Item  Is  being  tested  while  the  other  Is  being  fixed. 

50.3.2.2.3  RDGT  must  correct  failures  that  reduce  operational  effectiveness,  and  failures 
that  drive  maintenance  and  logistic  support  cost.  Therefore,  failures  must  be  prioritized 
for  correction  in  two  separate  categories;  mission  criticality,  and  cumulative  ownership  cost 
criticality.  The  differences  between  required  values  for  the  system  reliability  parameters 
shall  be  used  to  concentrate  reliability  engineering  effort  vrtiere  It  Is  needed  (for  example: 
enhance  mission  reliability  by  correcting  mission-critical  failures;  reduce  maintenance  man¬ 
power  cost  by  correcting  any  failures  that  occur  frequently). 

50.3.2.2.4  It  is  Imperative  ant  RDGT  be  conducted  using  one  or  two  of  the  first  full-scale 
engineering  development  Items  available.  Delay  forces  corrective  action  Into  the  formal 
configuration  control  cycle,  which  then  adds  even  greater  delays  for  admlnstratlve  processing 
of  reliability  engineering  changes.  The  cumulative  delays  create  monumental  retrofit  prob¬ 
lems  later  In  the  program,  and  may  prevent  the  Incorporation  of  necessary  design  corrections. 
An  appropriate  sequence  for  RDGT  would  be:  (1)  ESS  to  remove  defects  In  the  test  items  and 
reduce  subsequent  test  time,  (2)  environmental  testing  such  as  that  described  In  MIL-STD-810, 
and  (3)  combined-stress,  life  profile,  test-analyze-and-fix.  This  final  portion  of  RDGT 
differs  from  RQT  in  two  ways:  RDGT  Is  Intended  to  disclose  failures,  while  RQT  Is  not;  and 
ROGT  Is  conducted  by  the  contractor,  while  RQT  must  be  Independent  of  the  contractor  if  at  all 
possible. 
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Table  4-9  has  been  extracted  from  the  MIL-STD-785  Application  Guidance 
Section,  The  key  point  to  notice  Is  the  difference  in  purpose  of  the  RDGT 
and  RQT,  "RDGT  is  intended  to  disclose  failures;  and  P.QT  is  not"  and 
"testing  does  not  improve  reliability,  only  corrective  actions  that  pre¬ 
vent  the  recurrence  of  failures  in  the  operational  inventory  actually 
improve  reliability."  It  should  also  be  highlighted  that  "RDGT  Is  a 
planned,  prequal  if  ication,  tc.st-analyze-and-fix  process. "  For  complete - 


ness  in  differentiating  RDGT  from  RQT,  the  MIL-STD-785  application  guid¬ 
ance  with  respect  to  Task  303  RQT  has  also  been  included  as  Table  4-10.  It 
should  be  noted  that  there  are  no  data  item  descriptions  specifically 
associated  with  reliability  growth/TAAF  testing  although  DI-R-7033  "Relia¬ 
bility  Test  Plan,"  DI-R-7035  "Reliability  Test  and  Demonstration  Plan"  and 
DI-R-7034  "Reliability  Test  and  Demonstration  Reports"  cover  this  area. 
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TABLE  4-10:  MIL-STD-785B  RELIABILITY  QUALIFICATION  TEST 
APPLICATION  GUIDANCE 


50.3.3.1  Reliability  qualification  test  (RQT)  (task  303).  RQT  Is  Intended  to  provide  the 
government  reasonable  assurance  that  nnmmun  acceptable  reliability  requirements  have  been 
met  before  Items  are  comnltted  to  production.  RQT  must  be  operationally  realistic,  and  must 
provide  estimates  of  demonstrated  reliability.  The  statistical  test  plan  must  predefine 
criteria  of  compliance  ("accept")  which  limit  the  probability  that  true  reliability  of  the 
Item  Is  less  than  the  minimum  acceptable  reliability  requirement,  and  these  criteria  must  be 
tailored  for  cost  and  schedule  efficiency.  However: 

TESTING  TEN  ITEMS  FOR  TEN  HOURS  EACH  IS  NOT  EQUIVALENT  TO  TESTING  ONE  ITEM  FOR  ONE  HUNDRED 
HOURS,  REGARDLESS  OF  ANY  STATISTICAL  ASSUMPTIONS  TO  THE  CONTRARY. 

50.3.3.1.1  It  must  be  clearly  understood  that  RQT  Is  preproduction  test  (that  Is,  It  must 
be  completed  In  time  to  provide  management  Information  as  Input  for  the  production  decision). 
The  previous  concept  that  only  required  "qualification  of  tho  first  products  units*  meant 
that  the  government  committed  Itself  to  the  nroduction  of  unqualified  equipment. 

50.3.3.1.2  Requirements  for  RQT  should  be  determined  by  the  PA  and  specified  In  the 
request  for  proposal.  RQT  is  required  for  Items  that  are  newly  designed,  for  Items  that  have 
undergone  major  modification,  and  for  items  that  have  not  met  their  allocated  reliability 
requirements  for  the  new  system  under  equal  (or  more  severe)  environmental  stress.  Off-the- 
shelf  (government  or  cormiercial)  items  which  have  met  their  allocated  reliability  require¬ 
ments  for  the  new  system  under  equal  (or  more  severe)  environmental  stress  may  be  considered 
qualified  by  analogy,  but  the  PA  Is  responsible  for  ensuring  there  Is  a  valid  basis  for  that 
decision. 

50.3.3.1.3  Prior  to  the  start  of  RQT,  certain  documents  should  be  available  for  proper 
conduct  and  control  of  the  test.  Thece  documents  include:  the  approved  TEMP  and  detailed  RQT 
p-ocedures  document,  a  listing  of  tbc  items  to  be  tested,  the  Item  specification,  the 
statistical  test  plan  (50.3.1.6),  and  a  statement  of  precisely  who  will  conduct  this  test  on 
behalf  of  the  government  (50.3.1.7).  The  requirements  and  submittal  schedule  for  these 
documents  must  be  in  the  CDRL. 
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4.4.4  M1L-ST0-781C  "Reliability  Design  Qualification  and  Production 
Acceptance  Tests:  Exponential  Distribution"  (21  Oct  77)  (Currently  under 
revision  to  MIL-STD-781D,  see  paragraph  4.4.5):  This  document  in  Its 
present  form  does  not  address  reliability  growth  or  TAAF  testing.  It 
covers  RQT  and  PRAT.  Under  this  standard,  contractor  compliance  with 
numerical  reliability  is  determined  using  an  accept/reject  criteria  of  a 
specific  test  plan.  Corrective  actions  to  improve  the  system  reliability 
based  on  failure  occurrences  are  not  required. 

Although  TAAF  testing  is  not  covered,  the  standard's  example  of  a  time- 
phased  reliability  program's  activities  lists  TAAF  testing  as  an  FSED 
"Related  Task"  in  addition  to  the  RQT  as  a  "Key  Task."  The  standard  says 
with  respect  to  reliability  development  testing  "sufficient  testing  should 
be  conducted  to  provide  confidence  that  the  reliability  meets  or  exceeds  0Q 
(upper  test  MTBF).  This  is  a  test-analyze-and-f ix  (TAAF)  type  test  and 
normally  c°'n  '''ts  of  a  sequence  of  testing,  analyzing  all  failures,  incor¬ 
porating  'civ<  .ction,  and  retesting,  with  the  sequence  repeated 
until  as  ance  is  obtained  that  the  required  reliability  can  be  demon¬ 
strated  during  the  reliability  qualification  test."  On  the  other  hand, 
with  respect  to  RQT's  it  states  "reliability  qualification  tests  in 
accordance  with  MIL-STD-781  should  be  performed  to  provide  a  high  degree  of 
confidence  that  hardware  r "  bility  meets  or  exceeds  the  requirement." 

4.4.5  MIL-STD-781D  (31  Dec  80  draft):  Along  with  various  other 
changes,  this  draft  expanded  previous  edition  by  the  incorporation  of 
reliability  growth  testing.  The  draft  has  not  been  approved  and  the 


publication  of  MIL-STD- 1635(EC)  and  MIL-HDBK-189  have  caused  th?  scope  of 
MIL-STD-781D  to  be  reduced  In  the  reliability  growth  testing  area.  The  new 
draft  is  to  be  released  second  quarter  of  FY84. 


4.4.6  MIL-STl)-1635(EC)  "Rei i abi  1  ity  Growth  Testing11  (3  February  1978); 
"This  standard  covers  the  requirements  and  procedures  for  reliability 
development  (growth)  tests.  These  tests  are  conducted  during  the  ha^dwa:  = 
development  phase  on  samples  which  have  completed  environmental  tests 
prior  to  production  commitment,  and  ao  not  replace  othe^  tests  described  in 
the  contract  or  equipment  specification.  These  tests  provide  engineering 
information  on  failure  modes  and  mechanisms  of  a  test  item  under  natural 
and  induced  environmental  conditions  of  military  operations.  Reliability 
improvement  (growth)  will  result  when  failure  modes  and  mechanisms  are 
discovered  and  identified  and  their  recurrence  prevented  through  implemen¬ 
tation  of  corrective  action." 

"The  standard  is  applicable  to  Naval  Electronic  Systems  Command  procure¬ 
ments  for  development  of  all  systems  and  equipment  subject  to  contract 
definition  and  to  the  development  of  other  systems  and  equipment  when 
specified  in  the  equipment  specification." 

The  document  allows  the  contractor  to  determine  the  reliability  growth 
test  subject  to  procuring  activity  approval.  His  model  should  be  one 
"based  cn  previous  development  programs  -  for  systems/equipment  of  the 
same  type."  Unless  otherwise  specified,  it  requires  the  use  of  the  Duane 
Model.  The  performance  level  of  the  test  item  is  established  prior  to  the 
start  of  testing.  It  calls  for  a  fixed  length  period  of  testing  to  be 
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approved  by  the  procuring  activity  ind  states  that  5-25  multiples  of  the 
required  MTBF  will  generally  provide  sufficient  time  for  the  desired 
growth.  The  standard  states  that  the  "probable"  range  of  Duane  growth 
rates  is  between  0.3  and  0.6. 

In  terms  of  assessment,  the  standard  says  "as  long  as  the  achieved  reliabi¬ 
lity  growth  corresponds  favorably  with  the  planned  growth,  as  presented  in 
the  reliability  growth  test  plan  procedures,  satisfactory  performance  may 
be  assumed."  Satisfactory  is  further  defined  as  any  one  of: 

"A.  The  plotted  MTBF  values  remain  on  or  above  the  planned  growth 
line. 


3.  The  best-fit  straight  line  is  congruent  with  or  above  the  planned 

line. 


C.  The  best-fit  straight  line  is  below  the  planned  line  but  its  slope 
is  such  that  a  projection  of  the  line  crosses  the  horizontal  required  MTBF 
line  by  the  time  that  the  planned  growth  line  reaches  the  same  p,  int." 

An  important  point  to  be  made  regarding  failure  counting  is  that  the 
cumulative  MTBF  to  be  plotted  is  calculated  based  on  all  failures.  "This 
plot  shall  not  be  adjusted  b.y  negating  past  f a i  1 » ires  because  of  present  or 
future  design  changes. " 

The  standard  offers  an  alternative  moving  average  technique  for  relia¬ 
bility  assessment  and  states  MTBF  estimation  will  be  in  accordance  with 


M1L-STD-781.  It  suggests  "a  successful  reliability  growth  test  program 
may  result  in  the  deletion  of  reliability  demonstration  tests  if  reliabi¬ 
lity  requirements  are  fully  achieved  prior  to  production  commitment. 

The  standard  concludes: 

"Failure  to  provide  the  time  and  dollar  resources  necessary  for  reli¬ 
ability  growth  is  an  error  committed  much  too  often  in  research, 
development,  test  and  evaluation  planning." 

4.4.7  MIL-STD-2068  "Reliability  Development  Testing"  (21  March  1977): 
"This  standard  established  requirements  and  procedures  for  a  reliability 
development  test  to  implement  the  MIL-STD-785  requirement  for  such  a  test. 
The  purpose  of  the  reliability  development  test  is  reliability  growth  and 
assessment  to  promote  reliability  Improvement  of  systems  and  equipment  in 
n  ordinary  and  standarized  manner.  This  standard  is  applicable  to  Naval 
Air  Systems  Command  procurements  for  development  of  systems  and  equipment. 
The  reliability  development  tests  do  not  replace  the  design,  qualifica¬ 
tion,  or  other  required  tests  specified  for  the  systems  or  equipment." 

Regarding  establishment  of  a  pretest  performance  baseline,  the  standard 
states  "unless  otherwise  specified  prior  to  conducting  any  test,  the  test 
item  shall  be  tested  and  a  record  shall  be  made  of  all  data  to  determine 
compliance  with  required  performance."  Regarding  reliability  assessment 
it  states  "a  plot  of  achieved  reliability  expressed  as  a  point  estimate 
shall  be  used  to  depict  the  results  of  the  reliability  growth  test.  This 
plot  shall  be  made  showing  the  cumulative  reliability  versus  cumulative 
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test  time.  This  plot  shall  not  be  adjusted  by  negating  past  failures 
because  of  present  or  future  design  changes."  The  standard  calls  for  the 
presentation  of  a  second  "Adjusted  Reliability"  curve  to  depict  the  level 
at  which  the  achieved  reliability  would  be  if  these  failures  were  dis¬ 
counted  for  which  acceptable  corrective  action  has  resolved  a  failure  to 
the  satisfaction  of  the  procuring  activity."  With  respect  to  test  time,  it 
states  "unless  otherwise  specified,  when  two  or  more  test  items  are  used, 
the  minimum  operating  time  for  each  test  item  shall  be  not  less  than  one 
half  the  average  operating  time  for  all  items  on  test."  It  further  states 
"the  reliability  development  test  should  be  planned  as  a  fixed  length  test 
and  the  test  duration  must  be  specified.  Fixed  length  tests  of  10-25 
multiples  of  the  specified  MTBF  will  generally  provide  a  test  length  suf¬ 
ficient  to  achieve  the  desired  reliability  growth  for  equipment  MTBF's  in 
the  50  to  2000  hours  range.  For  equipment  MTBF's  over  2000  hours,  test 
lengths  should  be  based  i  equipment  complexity  and  the  needs  of  the 
program,  but  as  a  minimum,  should  be  one  multiple  of  the  specified  MTBF. 
In  any  event,  the  test  length  should  not  be  less  than  2000  hours  or  more 
than  10000  hours."  The  standard  supersedes  Aeronautical  Requirements 
documents  AR-104,  AR-108  and  AR-111  through  AR-118  which  addressed 
reliability  development  testing  for  specific  types  of  systems. 

4.4.8  MIL-HDBK-189  "Reliability  Growth  Management"  (13  February  1981): 
"This  handbook  provides  procuring  activities  and  development  contractors 
with  understanding  of  the  concepts  and  principles  of  reliability  growth, 
advantages  of  managing  reliability  growth  and  guidelines  and  procedures  to 
be  used  in  managing  reliability  growth." 


Methods  are  presented  for  planning,  evaluating  and  controlling  reliability 
growth.  It  states  "reliability  growth  management  is  part  of  system  engi¬ 
neering  procedures  (MIL-STD-497) .  It  does  not  take  the  place  of  other 
reliability  program  activities  (MIL-STD-785)  such  as  prediction  (MIL-STD- 
756),  apportionment,  FMEA  and  stress  analysis.  Instead,  reliability 
growth  management  provides  a  means  of  viewing  all  the  reliability  program 
activities  in  an  integrated  manner." 

Rather  than  the  monitoring  of  reliability  program  tasks  in  a  subjective 
manner,  reliability  growth  management  provides  a  quantitative  means  of 
making  timely  program  decisions  regarding  schedule  and  funds. 

Different  concepts  of  continuous  and  phase-by-phase  reliability  growth  ere 
discussed  as  they  apply  to  planning  and  tracking  a  program.  The  different 
approaches  of  implementing  of  design  "fixes"  and  tiie  risks  associated  with 
them  are  discussed.  Emphasis  is  on  applying  growth  techniques  on  a  phase- 
by-phase  basis.  Tracking  methodology  addresses  assessing  the  demonstrated 
reliability  as  well  as  the  projected  reliability.  The  projected  reliabi¬ 
lity  "serves  the  basic  purpose  of  quantifying  the  present  reliability 
effort  relative  to  the  achievement  of  future  milestones." 

The  planning  for  reliability  growth  is  addressed  on  a,phase-by-phase  basis 
and  statistical  tests  are  presented  for  determining  whether  growth  is 
occurring.  With  respect  to  models  the  handbook  says  "generally  speaking, 
the  simplest  model  which  is  realistic  and  justifiable  from  previous  exper¬ 
ience,  engineering  consideration,  goodness  of  fit,  etc.,  will  probably  be 
a  good  choice." 
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The  document  details  a  "how  to"  approach  for  contracting  for  reliability 
growth  including  what  should  be  in  the  request  for  proposal,  the  contrac¬ 
tor's  proposals  and  the  contract.  Planning,  testing  and  tracking  provi¬ 
sions  are  addressed.  With  respect  to  failure  purging,  the  handbook  is 
quite  explicit: 


"Failure  purging  as  a  result  of  design  fixes  is  an  unnecessary  and 
unacceptable  procedure  when  applied  to  determining  the  demonstrated 
reliability  value.  It  is  unnecessary  because  of  the  recently  devel¬ 
oped  statistical  procedures  to  analyze  data  whose  failure  rate  is 
changing.  It  is  unacceptable  for  the  following  reasons: 


The  design  fix  must  be  assumed  to  have  reduced  the  probabi¬ 
lity  of  a  particular  failure  to  zero.  This  is  seldom,  if 
ever,  true.  Usually  a  fix  will  only  reduce  the  probability 
of  occurrence;  and  in  some  cases,  fixes  have  been  known  to 
actually  increase  the  probability  of  a  failure  occurring. 


It  must  be  assumed  that  the  design  fix  will  not  interact  with 
other  components  and/or  failure  modes.  Fixes  have  fre¬ 
quently  been  known  to  cause  an  increase  in  the  failure  rate 
of  other  components  and/or  failure  modes." 


Further  rationale  is  presented  by  "if  there  has  been  sufficient  testing  to 
establish  the  effectiveness  of  a  design  fix,  then  an  appropriate  reliabi¬ 
lity  growth  model  will,  by  then,  have  sufficient  data  to  reflect  the  effect 
of  the  fix  in  the  current  reliability  estimate." 
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The  document's  appendices  present  a  variety  of  continuous  and  discrete 
reliaDility  growth  models  but  the  AMSAA  model  is  the  one  recommended  as 
"the  most  versatile  for  tracking  growth."  An  entire  detailed  appendix  is 
devoted  to  applying  the  AMSAA  model  including  parameter  estimation,  confi 
dence  interval  calculation,  and  goodne~s  of  fit  tests  for  the  three  failurt 
data  types;  time  terminated  testing,  failure  terminated  testing,  and 
grouped  data.  With  regard  to  the  type  of  failure  data  preferred  it  states: 
"In  general,  time  to  failure  data  are  preferred  over  data  in  which  the  time 
of  each  failure  is  unknown  and  all  that  is  known  is  the  number  of  failures 
that  occurred  in  each  period  of  time  (grouped  da' a).  Time  to  failure  data 
will  obviously  provide  more  information  for  estimating  system  reliability 
and  growth  rates." 

5.0  Reliability  Growth  Analysis:  If  the  concept  of  reliability  improve¬ 
ment  by  test,  detection  of  failure  causes,  and  design  changes  to  eliminate 
these  causes  is  accepted,  means  must  be  considered  for  planning  this  pro¬ 
cess,  assessing  the  current  status,  and  projecting  future  results.  A 
number  of  types  of  models  have  been  postulated  to  enable  these  goals  to  be 
accomplished.  While  the  intent  of  this  report  is  not  to  be  a  complete 
tutorial  on  analysis  techniques,  to  be  complete,  an  overview  must  be 
included. 

5.1  Reliability  Growth  Model  Types:  Reliability  Growth  Models  are  gener¬ 
ally  categorized  as  statistical  or  probabilistic  models  (Ref  43): 


Probabilistic  Models  -  Because  no  unknown  parameters  are  associated  with 


these  models,  the  data  obtained  during  programs  cannot  be  incorporated  and 
make  this  type  of  limited  use. 

Statistical  Models  -  Unknown  parameters  are  associated  with  these  models, 
in  addition,  these  parameters  are  estimated  throughout  the  development  of 
the  product  in  question. 

Another  way  of  distinguishing  among  models  is  whether  they  are  parametric 
or  not,  where  parametric  models  imply  there  is  a  pattern  to  the  growth. 
Nonparametric  models  allow  the  growth  curve  to  fall  where  it  will.  Some 
models  are  based  on  the  assumption  of  a  particular  failure  distribution, 
such  as  exponential.  Another  distinction  is  whether  a  model  is  continuous 
or  discrete.  In  general,  the  discrete  models  are  useful  for  reliability 
tests  which  involve  repeated  trials.  Continuous  models  tend  to  be  used 
more  in  cases  where  the  equipment  is  operated  until  failure  and  then 
repaired. 

An  Army  report  (Ref.  74)  described  a  different  classification  of  reliabi¬ 
lity  growth  models  as: 

A.  Deterministic  models  are  ones  in  which  the  precise  form  of  the 
reliability  growth  curve  is  known  for  a  particular  development  program  and 
system  before  development  is  initiated.  Consequently,  the  parameters 
associated  with  a  deterministic  model  are  fixed  by  the  model  user  prior  to 
any  development  effort. 


B.  Parametric  models  are  ones  that  utilize  early  growth  patterns 
exhibited  by  the  system  to  project  reliability  through  later  stages  of 
development. 

C.  Bayesian  models  assume  that  related  parameters  are  random  vari¬ 
ables  governed  by  appropriate  probability  density  functions.  Whereas 
parametric  techniques  utilize  recorded  test  data  to  estimate  model  para¬ 
meters,  Bayesian  models  employ  statistical  distributions  of  the  para¬ 
meters,  as  well  as  available  test  data. 

D.  Special  models  are  those  that  don't  exhibit  the  distinguishing 
features  of  the  previous  classifications. 

Table  5-i  summarizes  a  comparative  analysis  of  models  classified  in  the 
USAMC  study. 


MOOEL 


TYPE 


INPUT 

(REQUIRED  TEST  DATA) 


OUTPUT 

(RELIABILITY  INDICATOR) 


PROJECTIVE 

CAPABILITY? 


DUANE 

DETERMINISTIC 

TIMES-TO-FAILURE 

MtAN-TIME-TO-FAILURE 

YES 

LLOYD  &  LIPOW 
HYPERBOLIC 

PARAMETRIC 

SUCCESS-FAILURE 

DATA  FOR  EACH  BLOCK 

OF  TEST  TRIALS 

PROBABILITY  OF  SYSTEM 
SUCCESS  DURING  THE 

NEXT  TESTING  BLOCK 

YES 

LLOYD  &  LIPOW 

TWO- STATE 

PARAMETRIC 

NA 

PROBABILITY  OF  SYSTEM 
SUCCESS  DURING  THE 

NEXT  TEST  TRIAL 

YES 

WEISS 

PARAMETRIC 

TIMES-TO-FAILURE 

WITH  RESTRICTION  ON 
MAXIMUM  TIME 

MEAN-TIME-TO-FAILURF 

YES 

VIRENE 

PARAMETRIC 

ANY  CONSISTENT 

MEASURE  OF 

RELIABILITY 

ANY  CONSISTENT 

MEASURE  OF 

RELIABILITY 

YFS 

CHERNOFF  &  WOODS 

PARAMETRIC 

NUMBER  OF  SUCCESSES 
BETWEEN  CONSECUTIVE 
TRIAL  FAILURES 

PROBABILITY  OF  SYSTEM 
SUCCESS  DURING  THE 

NEXT  TEST  TRIAL 

CORCORAN  AND 

REED  EXTENSION 
MUST  BE  USED 

POLLOCK 

BAY  ESI  Af: 

TIME-TO-FAILURE 

OR  SUCCESS-FAILURE 

DATA  FOR  EACH  TRIAL 

MTBF  OR  PROBABILITY 

OF  SYSTEM  SUCCESS 

DURING  NEXT  TEST  TRIAL 

YES 

BARLOW  ft  SCHEUER 

SPECIAL 

SUCCESS-FAILURE 

DATA  FOR  EACH  8LOCK 

OF  TEST  -RIALS 

PROBABILITY  OF  SYSTEM 
SUCCESS  EXHIBITED  IN 
PREVIOUS  TESTING  BLOCK 

NO 

WOLMAN 

SPECIAL 

NA 

PROBABILITY  OF  SYSTEM 
SUCCESS  DURING  THE 

NEXT  TEST  TRIAL 

CORCORAN  AND 

REED  EXTENSION 
MUST  BE  USED 

5.2.1  The  Duane  Model:  Amonc  the  most  popular  models  for  reliabilty 
growth  is  the  Duane  Model.  In  1962,  J.T.  Duane  of  General  Electric  Com¬ 
pany's  Motor  and  Generator  Department  published  a  report  in  which  he  pre¬ 
sented  his  observations  during  development  programs  at  GE.  These  systems 
include  complex  hydromechanical  devices,  complex  types  of  aircraft  genera¬ 
tors  and  an  aircraft  jet  engine.  The  study  of  the  failure  data  was 
conducted  in  an  effort  to  determine  if  any  systematic  changes  in  reliabi¬ 
lity  occurred  during  the  development  programs  for  these  systems.  His 
analysis  revealed  that  for  these  systems  the  observed  cumulative  failure 
rate  versus  cumulative  operating  hours  closely  approximated  a  straight 
line  when  plotted  on  log-log  paper  (see  Figure  5.1).  Similar  plc-ts  have 
been  noted  in  industry  for  other  types  of  electrical  and  mechanical  sys¬ 
tems,  and  by  the  US  Army  for  various  military  weapon  systems  during 
development. 


FIGURE  5.1:  FAILURE  RATE  VERSUS  CUMULATIVE  OPERATING  HOURS  FOR  DUANE'S 
ORIGINAL  DATA 
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Duane's  postulate  was  that  as  long  as  re  liability- improvement  continues., 
his  mathematical  expression  would  hold  (Equ.  5.1). 


X_11B1  =  XT' 
cum 


(Equ.  5.1) 


MTBFCUM  =  K  J<X 


(Equ.  5.2) 


(Equ.  5.3) 


Xcum  =  cumulative  failure  rate 


T  =  cumulative  test  time  (Zt) 


F  =  total  number  of  failures  occurring  during  T 


K  =  constant  determined  by  the  initial  MTBF  and  the  initial 


conditioning  period 


a  =  growth  rate 


From  this  empirical  relationship  (Equ.  5.1)  the  cumulative  MTBF  can  be 
related  to  the  instantaneous  or  attained  MTBF  (MTBF  of  design  if  no 
further  design  changes  are  implemented)  as  follows: 


(From  Equ.  5.3) 


F  =  TKT"a 
F  =  KT(1'a) 


(Substituting  Xcum  =  KT"a) 


=  ( 1 -a)KT~a 


X(t)  =  (l-a)KT~a 


(Equ.  5.4) 


inst  K  ( 1 -a) 


(Equ.  5.5) 


^ 

;  \  s  \ 


Since  KT  a  is  the  cumulative  failure  rate  (rqu.  5.1),  Duane  concluded: 


*(t)  ■  (l-o.)  Xcum 


or 


MTBF 


_  MTBFcum 
inst  ( 1-a) 


(Equ.  5.6) 


For  many  systems,  the  plot  of  cumulative  MTBF  versus  cumulative  test  time 
is  a  straight  line  with  slope  alpha  (a),  when  plotted  on  log-log  paper.  If 
alpha  is  calculated  from  this  plot,  then  the  instantaneous  MTBF  may  be 
calculated  at  any  point  during  the  reliability  growth  program  using  Equ. 
5.6. 


Figure  5.2  shows  the  cumulative  MT3F  versus  cumulative  test  time.  The 

current  (or  instantaneous)  MTBF  is  drawn  parallel  to  the  cumulative  MTBF  on 

a  log-log  scale  and  has  a  value  of  -r-  X  MTBF„  . 

I -a  cum 


FIGURE  5.2:  DUANE  PLOT  FOR  RELIABILITY  GROWTH  OF  AN  AIRBORNE  RADAR 


In  order  to  plan  a  growth  test  or  to  predict  the  reliability  at  some  future 
time  the  model  parameters  a  and  K  must  be  known.  Depending  on  how  the 
model  is  being  used,  the  parameters  a  and  K  in  Equ.  5.1  may  be  determined 
by  one  particular  method  or  a  combination  of  methods  listed  below  in  order 
of  preference: 

A.  Historical  data  from  similar  systems  that  experienced  reliability 
growth. 

B.  Plot  initial  failure  data  on  log-log  paper  and  calculate  a  and  K 
when  a  linear  relationship  becomes  evident. 


W  - 


C.  Assign  a  and  K  based  on  an  engineering  analysis  and  on  manage¬ 
ment's  judgment  regarding  how  quickly  failures  may  be  revealed,  analyzed 
and  fixed. 

Methods  A  and  C  are  used  when  the  model  is  used  as  a  planning  tool  to  give 
management  an  idea  of  the  test  time  and  the  costs  of  implementing  a  relia¬ 
bility  growth  test. 

Method  B  is  used  when  the  model  is  used  as  a  tracking  tool  to  project  into 
future  time  whether  the  equipment  will  reach  its  goal  in  the  allotted  test 
time.  In  some  cases  up  to  1000  hours  of  test  time  is  needed  before  the 
characteristic  straight  line  is  observed.  This  is  shown  in  Figure  5.3  by 
the  initially  high  log  MTBF  decreasing  and  then  increasing  linearly  with 
log  time.  It  is  believed  that  this  initial  "hook"  in  the  Duane  plot  could 
resu it  from: 
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A.  An  initial  hook  in  the  bathtub  curve  as  shown  in  Figure  5.4  which 
would  give  an  early  high  MTBF  (low  failure  rate)  until  the  early  defects 
had  time  to  reveal  themselves.  This  may  indicate  that  the  equipment  is 
still  experiencing  a  burn-in  effect. 
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B.  The  unavoidable  reaction  time  before  the  effects  of  the  correc¬ 
tive  actions  begin  to  show  as  reliability  growth. 


FIGURE  5.3:  DUANE  PLOT  SHOWING  THE  INITIAL  "HOOK"  DURING  THE  EARLY  TIME 
PERIOD 


FIGURE  5.4:  INITIAL  HOOK  IN  BATHTUB  CURVE  SHOWING  AN  INITIALLY  LOW 
FAILURE  RATE  (HIGH  MTBF) 


/■/■.•Vi 

VWH 

O- 

‘  %.*'  \4 

v.-nov? 
4  *  *  __*•  s  *  -« 

Sis 

•  '.■\V.\v 

-  ■  'i 


^  ,-v 

x>> 


4 


51 


5  T>€  POINT  ESTIMATE  MTBF 
;»ING  THE  INDICATED  INTERVAL 


U 

100  200  300  400  MO  cOC  700  800  900  1000  1100  130  1300  1400  1500  16U0 
CUMULATIVE  TEST  TIME  (HOURS) 


An  example  of  parameter  estimation  and  growth  test  time  needed  is  given  in 
Section  6.3.3. 


The  Duane  parameters  a  and  K  can  also  be  determined  from  a  regression 

analysis  of  the  failure  data  using  equations  5.7  and  5.8. 

N  N  N 

E  (log  X.  log  M-)  -  (  E  log  X.  E  log  M.)/N 
i=l  1  1  i=l  1  i=l  1 


a 


Z  (log  X.)2  -  (  E  log  X.)2/N 
i=l  1  i=l  1 


(Equ.  5.7) 


log  p-  =  (  E  log  M. )/N  -  a  (  E  log  X.)/N  (Equ  5.8) 
i=l  i“l 


N 


N 


Where:  Xi  =  the  time  to  failure  of  failure  i. 


=  the  cumulative  MTBF  at  time  X^ . 


M  -  4*Ua  f  ^  1  f'l  I  m  k  .-\U»  A'f  ^  ^  lllMAa*  Ail  4-U  A  4-  A  .  4- 

m  one  ujtui  numuci  ui  iui  iuico  cntuuiaci  cu  uui  iny  one  i,cao. 


This  method  of  calculating  the  Duane  parameters  provides  better  accuracy 
than  graphical  techniques  and  can  easily  be  programmed  on  the  computer. 

5.2.2  The  AMSAA  Model:  Another  popular  model  is  the  AMSAA  reliability 
growth  model  which  is  more  complicated  than  the  Duane  model  but  enables  the 
calculation  of  statistical  goodness  of  fit  information  and  confidence 
limits.  For  a  more  extensive  treatment  of  this  model  the  reader  is 
referred  to  references  9,  28  and  53.  This  model  lends  itself  more  to 


tracking  reliability  growth  than  planning  growth  and  should  be  progranmed 
on  i.he  comouter  to  reduce  the  chance  of  error  during  the  long  calculations 
that  are  required. 

For  an  empirical  development  of  the  AMSAA  model,  the  Duane  postulate  given 
previously  is  considered.  Using  the  fact  that  the  plot  of  the  log  of  the 
cumulative  observed  failure  rate  (x'cum)  versus  the  log  of  time  is  a 
straight  line  leads  to  the  empirical  development  of  the  AMSAA  model.  Let¬ 
ting  primes  ('s)  denote  the  observed  quantities,  the  equation  of  this  line 
is: 


log  *'cum  =  K'  +  a'  log  T  (Equ.  5.9) 

Equating  X'cum  to  its  expected  (or  theoretical)  value  and  assuming  an  exact 
linear  relationship,  we  have: 


=  X, 


,og  Vcum  "  lcg  x 


cum 


Substituting  into  Equ.  5.9 


log  xcum  ■  K*  +  ct‘  log  T 


Taking  exponentials  gives 


X  =  e{K'+  1o9  T 

Acum 


a' ) 


X  =  eK'  Ta' 
cum 


Defining  XQ  =  eN  as  the  scale  parameter.  Since.  Xcum 


cumulative  failures  and  T  =  cumulative  test  time,  we  have 


I  =  X  Ta' 
T  Ao  1 


F  "  Ao  T 


a'  +  l 


Defining  3  =  a'+l,  as  the  shape  parameter 


F  =  X  r 
o 


(Equ.  5.10) 


The  instantaneous  failure  rate,  r{t),  of  the  system  is: 


r(t)  =  3T  =  V10"1  (Equ'  5'11} 

and  the  instantaneous  MTBF  is: 

MTBF.nt=  r(t)-1  -ill  (Equ-  5.12) 
l  ns t  -x  Q 


which  is  the  AMSAA  model. 


The  AMSAA  reliability  growth  model  assumes  that  system  failures  during  a 
development  testing  phase  follow  the  nonhomogeneous  Poisson  process  with 
Welbull  intensity  function  r(t)  =  Ao0T^”\  where  XQ>0,  0>O.  For  3  =  1, 
r(t)  =  XQ,  which  is  the  exponential  case.  For  0<1,  r(t)  is  decreasing, 
implying  reliability  growth.  For  3>1,  r(t)  is  increasing  indicating  a 
deterioration  in  system  reliability.  The  important  fact  to  note  is  that 
the  model  assumes  a  Poisson  process  with  Weibull  intensity  function  r(t)  s 
Xo0T^"\  and  not  the  W  ibull  distribution.  Therefore,  statistical  pro¬ 
cedures  for  Weibull  distribution  do  not  apply  for  this  model. 

A  comnon  sense  method  for  estimating  the  parameters  XQ  and  3  is  to  plot  the 
cumulative  number  of  failures  versus  cumulative  test  time  on  log-log  paper 
and  fit  a  line  to  these  points.  Aq  is  the  ordinate  of  the  line  correspond¬ 
ing  to  a  cumulative  test  time  of  one  hour  and  3  is  the  slope  of  the  line. 

An  improved  estimation  and  goodness  of  fit  procedure  has  been  developed  by 
Crow  (Ref.  9).  Using  the  result  that  the  plots  on  log-log  paper  imply  ’.hat 
the  successive  failure  times  of  a  system  follow  a  certain  stochastic  pro¬ 
cess  (i.e.,  the  nonhomogeneous  Poisson  process  with  Weibull  intensity 
XQ3tp  )  a  variety  of  useful  statistical  procedures  for  this  model  have 
been  derived. 

If  the  successive  times  of  failures  are  being  recorded  for  a  system  under¬ 
going  development  testing,  then  a  Cramer-von  Mises  statistical  goodness  of 
fit  test  can  be  performed  to  determine  if  the  AMSAA  reliability  growth 
model  is  appropriate.  If  the  model  is  acceptable,  then  maximum  likelihood 


55 


■r. 


(ML)  estimates  of  XQ  and  6  may  be  used  to  estimate  and  project  the  system 
MTBF.  Using  these  procedures  one  can  avoid  the  drawbacks  (no  confidence 
intervals  and  goodness  of  fit  measures)  associated  with  tracking  reliabi¬ 
lity  growth  from  log-log  plots.  Reference  53  presents  tables  for  confi¬ 
dence  intervals  and  critical  values  for  the  Cramer-von  Mises  equations 


that  apply  to  the  following  three,  types  of  data:  (1)  time  terminated  test 


data,  (2)  failure  terminated  test  data,  and  (3)  grouped  data.  For  these 


various  situations,  the  reader  is  referred  to  Appendix  C  of  reference  53 


for  in-depth  coverage  of  these  areas. 


It  should  be  noted  that  although  the  AMSAA  model  requires  all  failure  times 


for  estimating  the  parameters  X  and  B,  it  is,  in  effect,  a  self-purging 


model .  To  see  this,  let  8  be  the  estimate  of  8.  The  estimate  of  X  is  X  = 


/•s 

N/T^.  The  estimate  of  the  current  failure  rate  r(T)  =  X8T^~"*  is,  there- 
AA  Na  aTS-l  |N 

fore,  r(T)  -  XfBTp  =  y8  p  ~  j-  Note  that  N/T  would  be  the  failure 


rate  estimate  assuming  the  exponential  situation  of  no  growth.  However,  in 


tflP  nrp'lpnrp  nf  >-P  1  i  a  hi  1  H  t  v/  nrnuith  A  S  1  cp  that-  AM  «-  M  Tha  p  r +- ■!  +-o  t  T  ^ 
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using  tne  AMSAA  model  is  equivalent  to  using  the  exponential  method  but 
purging  (l-B)N  failures  arid  retaining  8N  failures. 


5.2.3  Duane  -  vs  -  AMSAA  Model:  The  Duane  model  is  often  expressed  as 
C(t)  =  Xt  a,  which  describes  the  same  pattern  of  growth  as  the  AMSAA  model 


when  a  -  8-1.  However,  the  Duane  model  considers  growth  to  be  determinis¬ 


tic,  while  the  AMSAA  model  gives  the  probabilistic  properties  describing 


the  growth  process.  The  probabilistic  nature  of  the  AMSAA  model  allows  a 


statistical  treatment  of  the  data.  Statistical  estimates  can  be  made  for 


assessment  purposes,  confidence  bounds  can  be  found,  and  the  data  can  be 
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subjected  to  an  objective  goodness -of-f it  test.  On  the  other  hand,  the 
deterministic  nature  of  the  Duane  model  is  particularly  suitable  for 
determining  the  planned  growth  curve  for  a  program. 

Some  practical  difficulties  in  applying  growth  models  are  listed  below: 

A.  The  parameter  estimates  are  dependent  on  how  much  test  time  has 
accumulated  before  they  are  calculated.  However,  the  parameters  need  to  be 
determined  early  in  a  growth  program  to  predict  future  reliability  and 
determine  if  the  requirement  will  be  met  within  the  allotted  test  time. 
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B.  The  plotting  methods  depend  on  the  subjective  appraisal  of 
whether  or  not  the  plotted  points  appear  to  lie  nearly  on  a  straight  line. 
The  best  fit  straight  line  is  sometimes  a  problem  because  of  the  tendency 
of  failures  to  bunch.  In  cases  of  difficulty,  less  importance  should  be 
attached  to  the  early  plots.  Green  (Ref  3)  has  found  that  instead  of 


plotting  as  each  failur 


it  is  better  to  do  so  after  time  intervals 


of  apo'CA imately  twice  the  target  MTBF.  However,  this  method  should  only 


be  used  within  systems  having  low  target  MTBF's. 


The  Duane  and  AMSAA  models  have  become  the  most  popular  because  of  their 
particular  advantages  as  follows: 


A  VA  A 'A  V- 


DUANE  MODEL 


►  ,  .  -A 


A.  It  is  mathematically  simple. 


B.  It  has  considerable  empirical  justification,  particularly  in 
development  of  electronic  hardware. 
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C.  The  parameter  a  is  directly  related  to  the  level  of  effort  of  the 


'  ’>  *  ’  *1 


reliability  program. 


D.  The  model  plots  as  a  straight  line  on  log-log  paper  allowing  for 
very  simple  illustration  of  the  reliability  growth  curve. 


AMSAA  MODEL 


A.  Its  probabilistic  nature  allows  a  statistical  treatment  of  the 
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5.2.4  Other  Models:  Although  the  Duane  and  AMSAA  models  are  the  most 


widely  used,  a  number  of  other  models  have  been  proposed  in  the  literature 
in  addition  to  those  already  mentioned.  Some  of  the  models  utilize  a 
continuous  time  scale,  others  utilize  a  discrete  time  scale,  implying  that 
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the  testing  is  performed  in  stages.  (Ref.  53)  provides  an  overview  of 
eight  discrete  and  nine  continuous  reliability  growth  models.  This  over¬ 


view  may  be  used  as  a  guide  for  choosing  a  candidate  model  for  a  particular 
appl  ication. 
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In  1975  Hughes  Aircraft,  under  contract  to  RADC,  performed  a  study  (Ref  10) 
of  the  applicability  of  six  reliability  growth  models  to  various  classes  of 
ground  based  and  airborne  systems  in  two  basic  environments: 

A.  "In-house"  where  failure  reporting  and  analysis  is  closely  con¬ 
trolled  and  corrective  actions  are  taken. 

B.  "In-field"  where  the  equipment  or  system  operates  in  its  intended 
use  environment  and  where  failures  are  reported. 

The  six  models  compared  (see  Ref  10  for  a  complete  model  description)  were: 

A.  Duane  Model 

B.  IBM  Model 

C.  Exponential-Single  Term  I’ower  Series  Model 

D.  Lloyd-Lipow  Model 

E.  Aroef  Model 

F.  Simple  Exponential  Model 
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Each  of  the  six  models  was  fitted  to  data  sets  (186  data  sets  for  ground 
equipment  and  84  for  airborne  equipment).  Most  of  the  study  data  was 
obtained  from  Hughes  built  systems;  however,  some  external  data  from  the 
Naval  Ship  Weapon  Systems  Engineering  Station,  Port  Hueneme,  California, 
was  obtained  for  ground  computers  and  displays.  Although  old  (1975),  its 
the  latest  comparison  of  model  fit  we  know  of.  Table  5-2  indicates  the 
types  of  equipment/systems  studied.  Table  5-3  provides  more  details  of  the 
equipment. 

TABLE  5-2:  RELIABILITY  GROWTH  STUDY  SYSTEM/EQUIPMENT  DESCRIPTIONS 


Shipboard  Radar 

Ground  Based  Radar 

Satellite  Microwave  Link 

Shipboard  Satellite  Microwave 
Communication 

Weapon  Control 

Radar  Display 

Computer 

Ground  Based  Radar 

Laser  Range  Finder 

Radar  Display  and  Computer 

Visual  Scan  System 

Laser  Bombing  System 

Airborne  Computer 

Infrared  System 

TABLE  5-3:  RELIABILITY  GROWTH  STUDY  EQUIPMENT  CATEGORIES 


1. 

Antenna 

Pedestal,  dish,  driver  gears,  motor, 
hydraulics 

2. 

Rad  ar 

Receiver,  exciter,  signal  processor, 
transmitter,  power  supplies 

3. 

Microwave 

Reciever,  exciter,  klystron,  transmitter, 
power  supplies 

4. 

Display 

CRT,  data  input  console,  display  controls, 
power  supplies 

5. 

Computer 

Computer  circuits,  CPU,  nr-ory,  power 
suppl ies 

6 

Comnunication 

Radio  receiver,  teletype,  etc. 

7 

System-Rad ar 

Complete  radar  system 

a. 

System-Mici  owavc- 

Complete  microwave  system 

9. 

System-Laser 

Complete  laser  system 

10. 

System- Infrared 

Complete  infrared  system 

11. 

System-Visual  Scan 

Complete  system  for  nighttime  sighting 

12. 

Laser  Transmitter 

Laser  transmitter  and  optics,  control 
electronics,  power  supplies 

13. 

Laser  Receiver 

Photo  diode  detector  and  optics 

14. 

Laser  Xmtr/Rcvr 

Laser  transmitter  and  receiver,  control 
electronics,  power  supplies 

15. 

Irfrared  Receiver 

IR  receiver  and  amplifier,  power  supplies 

In  addition  to  including  reliability  growth  information,  the  data  set  for 
each  equipment  also  included  information  relative  to  che  scope  of  the 
reliability  program  associated  with  that  equipment. 


SS 


‘j 


\  l' 

-i 


In  order  to  determine  the  degree  of  fit  of  the  models  to  the  data,  two 
goodness  of  fit  parameters  were  calculated,  R  and  R.E.  R  is  defined  as  the 
absolute  percentage  error  in  the  predicted  versus  the  observed  values. 
R.E.  measures  the  fraction  of  unexplained  variation  to  the  total 
variation.  The  smaller  the  values  of  R  and  R.E,  the  better  the  fit 
(ideally  R  =  R.E.  =  0).  Table  5-4  provides  a  comparison  of  the  models  in 
terms  of  fit  to  ground  and  airborne  equipment.  Table  5-5  provides  a 
comparison  of  models  by  equipment  category. 

TABLE  5-4:  RELIABILITY  GROWTH  STUDY:  JOINT  GOODNESS  OF  FIT  ANALYSIS 
FOR  AIRBORNE/GROUND  AND  IN-HOUSE  FIELD  CLASSIFICATIONS 


GROUND 

AIRBORNE 

IN-HOUSE 

FIELD 

— 

IN-HOUSE 

FIE 

LD 

R 

R.E. 

R 

R.E. 

R 

R.E 

R 

R.E. 

Duane 

28.64 

0.73 

24.38 

1.01 

25.44 

0.54 

67.88 

4.1373 

IBM 

23.43 

1.15 

26.85 

1.73 

23.96 

0.42 

13.66 

0.51 

Exponential 

24.41 

1.21 

32.05 

2.11 

11.41 

0.10 

7.38 

0.07 

Lloyd-Lipow 

25.32 

C.64 

20.35 

0.66 

28.42 

0.58 

11.79 

0.27 

Aroef 

22.30 

0.62 

19.21 

0.63 

23.70 

0.55 

10.57 

0.18 

Simple 

Exponential 

16.95 

0.36 

13.08 

0.35 

13. 76 

0.24 

12.20 

0.31 

The  following  conclusions  are  evident  from  Table  5-4: 

A.  The  Duane  Model  cannot  be  recommended  for  airborne  field  data. 

B.  Conversely,  the  IBM  model  is  excellent,  at  its  best,  for  airborne 
field  data. 
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DUANE 

IBM 

EXPONENTIAL 

Antenna 

35.9850 

1  .0482 

16.7530 

0.7259 

23.0410 

0.5796 

Radar 

20.0280 

0.4015 

50.1790 

1.7720 

72.3920 

6.2718 

Microwave 

19.0350 

0.7838 

25.44  30 
0.99C8 

15.4510 

0.6356 

Display 

28.4680 

1.1747 

24.8820 

0.7938 

33.C450 

1. 1845 

Computer 

28.5570 

1  3  TAT 

I  .  1  JO/ 

46.8850 

a  nnrr\ 

C» OOUW 

44.9850 

n  mnn 

L.JIUU 

Cofimunlcatlons 

30.7875 

2.4698 

19.5005 

0.8457 

30.8080 

0.9524 

System-Radar 

14.5100 

0.1688 

26.7090 

1.3847 

189.3860 

8.1803 

System- 

Microwave 

19.3220 

0.9852 

19.1505 

0.7591 

16.0805 

0.7144 

System-Laser 

19.3820 

0.7010 

219.9044 

2.3913 

8.2890 

0.0189 

System- Infrared 

65.9675 

4.2379 

14.2100 

0.5450 

11.6100 

0.1148 

System 

Visual  Scan 

13.4620 

0.2909 

44.3915 

1.6316 

8.7840 

0.1942 

Laser 

Transmitter 

33.6590 

0.2355 

138.9970 

0.6332 

15.6250 

0.0243 

Laser 

Receiver 

51.24.80 

0.3118 

126.7180 

0.9517 

12.0280 

0.0394 

Laser  Xntr/ 

Rcvr 

25.2970 

0.1163 

158.5713 

0.9805 

11.4100 

0.0293 

Infrared 

Receiver 


41.4885 

0.S573 


16.1805 

0.3365 


22.4500 

0.0816 


The  following  conclusions  can  be  drawn  from  Table  5-5: 


A.  For  antennas,  all  the  models  except  the  Duane  Model  are  quite 

good. 

8.  For  radar  an!  microwave  systems/equipment,  the  Duane  Model  and 
the  simple  exponential  model  are  very  good. 

C.  For  display,  computer  and  communications  equipment,  the  Lloyd- 
Lipow,  Aerof  and  simple  exponential  models  are  good. 

D.  For  infrared  systems  equipment,  all  models  but  the  Duane  are 
excellent. 

E.  For  all  laser  systems/equipment,  the  exponential  is  vastly  super¬ 
ior  to  all  other  models. 

F.  For  the  visual  scan  equipment,  the  exponential  model  is  again 
superior  to  the  remaining  models. 

G.  The  Duane  model,  while  rarely  fitting  "best"  was  seen  to  fit  in 
almost  all  the  cases. 

5.2.5  Nonrelevant  Failures:  Reference  56  presents  a  technique  for 
determining  the  learning  equation,  and  thereby,  for  predicting  nonrelevant 
failure  occurrences.  The  decrease  of  nonrelevant  failure  occurrences  over 
an  equipment's  life,  especially  those  due  to  infant  mortality,  is  a  result 
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of  a  learning  process  and  can  be  mathematically  predicted.  This  relation¬ 
ship  has  been  demonstrated  through  use  of  data  obtained  from  systems  com¬ 
posed  of  many  different  electronic  equipments. 

6.0  Reliability  Growth  Management  Techniques:  Reliability  growth  pro¬ 
grams  for  sophisticated  complex  systems  require  considerable  resources 
si  ch  as  time,  money  and  manpower  to  achieve  the  level  of  system  reliability 
acceptable  to  the  user.  During  the  growth  process,  the  total  system  or 
major  subsystems  are  tested  to  failure,  system  failure  modes  are  deter¬ 
mined  and  design  and/or  process  changes  are  implemented  to  eliminate  these 
modes  or,  at  least,  to  decrease  their  rate  of  occurrence.  If  this  process 
is  continued  and  design  and  process  modifications  are  made  in  a  competent 
manner,  then  the  system  reliability  will  increase. 

It  is  advantageous  for  the  program  manager  to  plan  and  track  this  increase 
in  system  reliability  during  the  development  program.  He  may  then  deter¬ 
mine  as  early  as  possible  whether  or  not  the  system  reliability  is  growing 
at  a  sufficient  rate  to  meet  the  required  goal  and  allocate  available 
resources  accordingly.  In  this  regard,  a  program  manager  needs  to  deter¬ 
mine  from  test  data  the  current  reliability  status  of  the  system,  estimate 
the  rate  of  growth,  and  obtain  projections  of  future  expected  reliability. 

Some  of  the  important  questions  that  need  to  be  addressed  in  planning  a 
reliability  growth  program  aie: 


A.  Is  a  growth  test  appropriate  for  this  program? 


B.  Is  the  final  reliability  objective  similar  to  reliability 
achievements  made  on  past  programs? 

C.  What  is  the  expected  starting  reliability  level  for  the  reliabi¬ 
lity  growth  curve  (e.g.,  10%  of  the  prediction)  3nd  how  many  hours  must  the 
equipment  be  preconditioned  before  this  starting  point  is  realized? 

0.  How  much  time  needs  to  be  allotted  for  grov.'th  testing? 

E.  How  many  units  should  be  allocated  to  reliability  testing  as  part 
of  the  overall  test  program? 

F.  What  minimum  test  time  should  be  required  on  each  unit  on  test? 

G.  What  milestones  for  reliability  growth  achievement  need  to  be 
established? 

It  must  he  stressed  that  the  answers  to  the  above  questions  are  not  "cook¬ 
book"  and  each  program  has  to  be  carefully  tailored  to  the  particular 
situation  and  the  particular  system. 

The  basic  tools  for  planning  a  reliability  growth  program,  and  thus  provid¬ 
ing  guidelines  to  amwer  the  aforementioned  questions,  are  discussed  in 
the  following  sections. 

6.1  Reliability  Growth  Test  or  Not:  The  costs  of  implementing  reliabi¬ 
lity  growth  into  a  contract  may  seem  excessive,  expecially  when  one  argues 
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that  a  cont  'actor  may  perform  an  Informal  growth  program  anyway  to  discover 
gross  design  errors.  However,  in  past  programs  discovery  of  noncompliance 
(reject  decision  in  the  reliability  qualification  test)  has  occurred  many 
times  after  full  scale  engineering  development.  Because  the  costs  of 
design  changes  are  more  expensive  the  later  they  are  implemented,  the 
customer  has  only  four  options  after  this  discovery,  none  of  which  is 
appetizing: 

A.  Accept  the  deficient  hardware,  which  means  added  life  cycle  costs 
because  of  additional  maintenance,  repair  and  logistics  actions  along  with 
lower  operational  availability. 

B.  Require  correction  of  defects,  which  means  accepting  added  delays 
and  costs. 

C.  Contract  to  another  supplier  for  an  equivalent  equipment,  which 
undoubtedly  involves  delays  and  costs  at  least,  as  great  as  option  (B)s 

D.  Cancel  the  entire  program. 

The  limited  customer  options,  together  with  the  historical  record  that 
shows  an  overwhelming  preference  for  option  (A)  indicates  that  the  threat 
of  failing  a  demonstration  test  if  design  problems  exist  may  be  no  threat 
at  all.  This  is  one  reason  that  the  costs  of  a  reliability  growth  program 
are  justified.  The  customer  is  not  only  buying  a  more  reliable  product, 
but  is  buying  visibility  to  guarantee  that  the  actual  status  of  reliability 


is  known  throughout  the  engineering  development  phase.  With  this  visibi¬ 
lity,  a  program  manager  can  assess  the  program's  reliability  status  and 
take  a  good  hard  look  at  why  the  reliability  milestones  are  not  being  met. 
By  doing  so,  he  is  in  a  position  to  redirect  resources  in  the  early  phases 
of  development  to  avoid  having  to  settle  for  one  of  the  four  options  listed 
above. 

The  most  cost  effective  way  to  grow  reliability  in  a  large  complex  system 
is  to  first  identify  low  reliability  equipment  via  a  prediction  and  then 
place  extra  emphasis  on  the  growth  programs  of  the  iow  reliability  equip¬ 
ment.  Fixed  length  tests  have  been  found  to  be  most  appropriate  for 
reliability  growth  in  terms  of  cost-effectiveness,  since  suppliers  faced 
with  testing  of  uncertain  duration  tend  to  protect  themselves  against 
worst  case  test  durations  ir,  their  pricing.  Cox  and  Keely  (Ref.  11)  have 
noted  that  in  many  successful  reliability  programs  using  reliability 
growth  philosophy,  approximately  40  to  50%  of  the  total  reliability  dollar 
was  allocated  for  growth  testing. 

The  program  manager  has  two  options  for  a  fixed  amount  of  reliability  test 
time.  The  op!  ons  are: 

1.  A  higher  reliability  level  through  more  growth  testing  at  a  cost  of 
less  time  for  demonstration  ,  and  thus  a  lower  confidence  in  demonstrated 
reliability. 

2.  A  higher  confidence  through  demonstration  testing  at  a  cost  of  less 
time  for  growth  testing,  and  thus  lower  achieved  reliability. 


These  options  are  shown  graphically  in  Figure  6.1. 


r I GURE  6.1: 


OPTIONS  AVAILABLE  TO  A  PROGRAM  MANAGER  FOR  A  FIXED 
RELIABILTY  TEST  TIME 
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Once  the  extent  of  testing  has  been  determined  through  a  review  of  the 
reliability  specified  and  its  relationship  to  the  state-of-the-art,  then 
evaluations  and  tradeoffs  should  be  made  to  determine  what  tests  to  include 
and/or  emphasize. 


The  nature  of  the  procurement  (i.e.,  new  development,  production,  off-the- 
shelf,  etc.)  will  dictate  to  a  large  extent  the  type  of  tests.  If  hardware 
to  be  procured  is  an  off-the-shelf  commercial  product,  PDGT  may  not  be 
appropriate  since  the  equipment  is  probably  mature  and  any  design  change 


would  be  difficult  to  obtain.  However,  if  the  off-the-shelf  equipment 
requires  complex  rfaces  then  RDGT  becomes  more  feasible.  Figure  6.2 
provides  son*  g.  ance  as  to  the  type  of  test  required  as  a  function  of 
contract  type  (Ref  23).  cor  example,  for  a  new  development  contract 
reliability  growth  testing  is  applicable  for  R&M  level  A  and  8  but  not 
level  C  (see  Table  6-9  for  application  levels). 

FIGURE  6.2:  RELIABILITY  TESTS  AS  A  FUNCTION  OF  CONTRACT  TYPE 
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A  -  HIGHEST  RELIABILITY  EFFORT 
C  •  LOWEST  RELIABILITY  EFFORT  (Sit  TABLE  6-9) 


Reference  21  presents  the  following  guidelines  on  when  a  Reliability 
Demonstration  Test  or  Reliability  Qualification  Test  (RQT)  is  most  cost 
effective.  A  demonstration  test  should  be  specified  only  if: 

A.  Demonstration  can  be  completed  sufficiently  parly  for  a  major 


redesign  cycle  and  timely  i ncorporation  into  production  hardware. 


B.  Realistic  incentives  and  penalties  are  defined  and  implemented 
for  reliability  achievement  or  failure. 

C.  The  customer  is  prepared  to  take  drastic  action,  up  to  contract 
cancellation,  to  enforce  reliability  and  schedule  guarantees. 

Obviously,  when  included  in  the  program  plan,  RQT  should  be  employed  selec¬ 
tively,  applied  only  to  those  specific  procurement  items  that  satisfy 
these  criteria. 

6.2  Planning  for  Reliability  Growth:  Initially,  one  wishes  to  depict  the 
generalized  growth  pattern  for  a  particular  class  of  systems  oeveloped 
utilizing  historical  data  on  similar  systems  and  equipments  and  develop¬ 
ment  programs  in  order  to  make  estimates  of  test  time  and  resources  needed. 
The  data  includes  expected  growth  rates  and  expected  initial  levels  of 
reliability.  System  characteristics  that  affect  growth  patterns  include 
challenge  to  the  state-of-the-art,  system  complexity,  the  nature  of  the 
system  (ground  or  airborne,  mechanical  or  electrical,  etc.)  along  with 
characteristics  of  the  development  program.  Other  characteristics  that 
affect  growth  patterns  are  test  facilities,  failure  analysis  capabilities 
and  management's  attitude  toward  a  growth  program.  Thus,  the  growth  rate 
is  not  only  a  function  of  the  type  of  equipment  being  built,  but  is  also 
depencant,  to  some  extent,  upon  the  company  performing  the  work.  Two 
different  approaches  are  commonly  used  in  the  analysis  of  historical  data 
and  the  development  of  planned  growth  curves.  The  more  traditional 
approach  has  been  to  treat  the  entire  development  program  as  in  idealized 
(smooth)  process.  The  other  approach  treats  the  development  program  as  a 
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phase-by-phase  process.  Figure  6.3  illustrates  the  sU  •’volved  for 
planning  a  growth  program  with  continuous  fixes  implemented  during  the 
program.  A  similar  procedure  is  used  for  planning  a  phase-by-phase  type 
test  (Figure  6.4) . 


FIGURE  6.3:  PLANNEO  RELIABILITY  GROWTH  (CONTINUOUS) 
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FIGURE  6.4:  PLANNEO  RELIABILITY  GROWTH  (PHASE -BY-PHASE) 
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In  analyzing  historical  data  for  planning  purposes,  care  should  be  exer¬ 
cised  to  assure  that  the  parameter  values  are  those  for  the  system  config¬ 
uration  that  was  being  tested  and  not  theoretical  values  for  some  hypothet¬ 
ical  "paper"  configuration. 

When  the  "delayed  method"  (of  implementing  fixes)  is  used,  the  growth  rate 
a  will  be  much  smaller  than  it  would  be  using  the  continuous  method.  This 
is  because  most  of  the  growth  occurs  between  test  phases  rather  than  during 
them.  One  problem  with  this  approach  is  that  neither  the  Duane,  the  AM5AA, 
nor  any  other  model  predicts  the  magnitude  of  the  jump  in  reliability  from 
one  phase  to  another.  However,  with  the  continuous  method,  the  test  has  to 
be  stopped  for  every  failure  and  the  cost  of  tying  up  test  resources  while 
waiting  for  failure  analysis  and  design  changes  is  prohibitive,  making  the 
delayed  method  more  practical.  One  should  plan  what  method  (delayed  fixes 
or  continuous  implementation  of  fixes)  will  be  used.  A  mixture  of  methods 
can  also  be  used,  for  example,  if  a  corrective  action  is  obvious  and  can  be 
taken  in  a  timely  manner,  then  the  test  can  be  stopped  and  the  fix  imple¬ 
mented;  however,  if  no  obvious  corrective  action  can  be  found,  then  for 
practical  reasons,  an  in-depth  failure  analysis  must  begin  and  the  fix 
implemented  at  the  end  of  a  test  phase  or  as  soon  as  a  corrective  action 
becomes  available.  In  many  cases,  where  the  delayed  method  is  used,  an 
additional  equipment  is  made  available  to  go  on  test  when  a  failure  occurs. 

If  the  planned  test  time  will  take  too  much  calendar  time,  then  more  than 
one  equipment  must  be  put  on  test.  If  this  is  the  case,  then  one  must  take 
into  account  how  many  equipments  will  fit  into  one  chamber  and  how  many 
chambers  must  be  available  for  the  test  in  addition  to  how  many  work  shifts 
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a  test  must  run  in  order  to  keep  a  program  on  schedule.  Reference  38 
described  an  overall  test  efficiency  (defined  as  a  ratio  of  weekly  accumu¬ 
lated  relevant  test  hours  to  possible  relevant  test  hours)  and  found  it  to 
be  approximately  50  percent.  Contributing  to  the  inefficiency  of  testing 
are  delays  associated  with  definition  of  corrective  actions,  lack  of  test 
articles  to  replace  equipment  in  troubleshooting  and  repair,  and  downtime 
for  repair  of  test  equipment. 

In  some  cases,  jumps  in  reliability  associated  with  delayed  fixes  are 
negative  (dips)  as  shown  in  Figure  6.5.  This  situation  often  occurs  at 
such  times  as  the  beginning  of  low  rate  production  when  the  manufacturing 
process  is  in  the  early  stages  of  a  "production  learning  curve."  A  new 
production  reliability  growth  process  must  then  take  place  to  regain  pre- 
production  reliability.  "Dips"  may  also  be  caused  by  new  problems  that 
crop  up  with  a  design  change  to  fix  some  other  problem  or  by  equipment 
interface  problems  if  the  initial  testing  is  not  performed  on  the  complete 
system  configuration. 


6.2.1  Initial  Reliability:  The  starting  point  represents  an  initial 
value  of  reliability  for  the  newly  developed  hardware  and  usually  falls 
within  the  range  of  10  to  40  percent  of  the  inherent  or  predicted  reliabi¬ 
lity  after  some  preconditioning  period.  Estimates  of  the  starting  point 
can  be  derived  from  prior  experiences  or  based  on  percentages  of  the 
estimated  inherent  reliability.  Historical  data  should  be  used  whenever 
possible;  however,  if  no  prior  data  is  available  from  a  similar  system  then 
a  commonly  used  estimate  of  10  to  20  percent  of  the  predicted  reliability 
can  be  used.  Starting  points  must  take  into  account  the  amount  of  reliabi¬ 
lity  control  exercised  during  the  design  program  and  the  relationship  of 
the  system  under  development  to  the  state-of-tne-art.  Higher  starting 
points  minimize  test  time.  It  should  be  noted  that  the  starting  point 
reliability  applies  to  the  system  after  preconditioning  that  allows  the 
data  to  "settle  down."  This  means  that  the  preconditioning  period  is 
unplotted,  but  since  the  basic  plot  is  cumulative  MTBF,  the  data  accumu¬ 
lated  during  this  initial  period  do  influence  later  results. 

Other  types  of  development  programs,  particularly  those  for  mechanical 
systems,  may  not  have  as  extensive  an  historical  data  base  to  draw  upon. 
In  those  cases,  starting  points  can  be  based  on  advanced  development  proto¬ 
type  test  data  or  on  synthesis  of  component  and  subsystem  results. 

6.2.2  The  Growth  Rate  (a) 

The  growth  rate,  which  is  the  slope  of  the  growth  curve,  is  governed  by  the 
amount  of  control,  rigor  and  efficiency  by  which  failures  are  discovered, 
analyzed  and  corrected  through  design  and  quality  actions.  A  large  value 
of  a  (a>0.5)  reflects  a  hard-hitting,  aggressive  reliability  program  with 


management  support  spanning  all  functions  <  r  a  knowledgeable  organization, 
while  a  low  value  of  a  (a<  C.l)  reflects  the  growth  in  reliability  that  is 
due  largely  to  the  need  to  resolve  obvious  problems  that  impact  production, 
and  to  implement  corrective  action  resulting  from  user  experience  and 
complaints.  Green  (Ref  3)  noted  that  a  high  growth  rate  (a)  does  not 
necessarily  indicate  a  good  design  as  is  often  thought,  but  it  does  show  a 
very  thorough  effort  by  the  whole  organization  and  particularly  by  the 
reliability  engineers,  to  discover  the  cause  of  the  failures  and  eliminate 
them.  In  fact,  with  excellent  design  and  manufacture  a  could  approach 
zero.  Negative  growth  can  sometimes  be  observed  when  engineering  changes 
are  implemented  to  improve  "performance,"  at  the  risk  of  loss  in  reliabi¬ 
lity.  The  maximum  value  of  ci  that  can  be  expected  is  not  greater  than  0.7 
because  of  the  lag  time  associated  with  revealing  failures,  analyzing 
them,  and  implementing  corrective  actions.  In  many  growth  programs  a 
ranges  from  .3"  to  .5  as  shown  in  Tables  6-1,  6-2,  and  6-3  (Ref  34)  which 
show  the  variation  of  growth  rates  from  in-service  use  improvement  pro¬ 
grams,  development  tests,  and  reliability  improvement  warranties.  Table 
6-4  summarizes  the  data  showing  that  the  effectiveness  of  a  growth  effort 
as  a  function  of  time,  with  the  development  phase  growth  effort  the  most 
beneficial . 

Herd  (Ref  34)  found  that  the  mean  growth  rate  for  a  large  electronic  system 
with  a  single  program  manager  that  placed  considerable  emphasis  on  devel¬ 
opment  testing  and  had  different  subcontractors  for  the  component  systems 
was  0.41,  with  a  standard  deviation  of  0.20. 


Cod i er  (Ref  1)  presented  some  general  observations  pertaining  to  growth 
rate  values.  They  are  that  the  growth  rate  (a)  is  higher: 

A.  For  analog  hardware  than  for  digital  hardware. 

B.  For  equipment  of  low  maturity  than  in  production  hardware. 

C.  In  equipment  exposed  to  severe  test  conditions  than  in  equipment 
undergoing  bench  tests. 

D.  In  proportion  to  the  hardware  oriented  reliability  improvement 
effort. 

The  differences  in  growth  parameters  observed  in  the  various  programs 
reflect  the  amount  and  timeliness  of  critical  engineering  information 
available  for  corrective  action  determination  and  the  nature  of  the  system 
(Mechanical,  Electronic,  etc.). 

The  Analytical  Sciences  Corporation  (TASC),  under  contract  to  RADC,  is 
developing  methodology  for  predicting  reliability  growth  characteristics 
as  a  function  of  equipment  attributes  and  program  characteristics.  The 
results  will  be  available  as  a  decision  and  planning  tool  around  April 
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TABLE  6-1 


RELIABILITY  GROWTH  RATES  FOR  ELECTRONIC  EQUIPMENT  FROM 
IMPROVEMENT  PROGRAMS  DURING  SERVICE  USE 


EQUIPMENT 


Airborne  Teletypewr iter 
Airborne  Radar  Altimeter 


OBSERVED 

a-VALUE 


Airborne  Search  Radar 


+0.01 


Airborne  Computer  Recorder 
Airborne  HF  Comnuni  cat  ions 


+0.11 

+0.12 


Airborne  UHF  Communications 


+0.13 


Airborne  Navigation  Set 
Shipborne  Acquisition  Radar 
Shipborne  Data  Processor 
Airborne  Radio  Navigation 
Airborne  Sonobuoy  Receiver 
Airborne  Tactical  Data  Display  (A) 
Airborne  Radar  Scan  Converter 
Airborne  Tactical  Data  Display  (B) 


+0.14 


+0.14 

+0.17 

+0.19 

+0.19 


+0.19 


+0.23 

+0.24 


Airborne  Inertial  Navigation 


+0.30 


TABLE  6-2:  RELIABILITY  GROWTH  RATES  OBSERVED  FOR  DIFFERENT  HAkDWARE 

SYSTEMS  IN  DEVELOPMENT  TESTS 


ITEM 

OBSERVED 

a-VALUE 

Gatling  Type  AA  Gun 

+0.40 

Hydro-Mechanical  Devices 

+0.49 

Pulse  Transmitter,  Radar 

+0.35 

Continuous  Wave  Transmitter 

+0.35 

Aircraft  Generators 

+0.39 

Analog  Receivers 

+0.49 

Airborne  Radar 

+0.48 

Airborne  Radar  (UK) 

+0.43 

Digital  Computer 

+0.48 

Jet  Engines 

+0.35 

High-Power  Equipment  (Power 

Supply,  Microwave  Amps) 

+0.30 

Satellite  Comm.  Terminal 

+0.34 

Modem  (Digital  Comm.  Terminal) 

+0.29 

TABLE  6-3:  EXAMPLES  OF  RELIABILITY  GROWTH  RATES  UNDER  RIW  PROGRAMS 


ITEM 

PLANNED 

ct-VALUE 

ACTUAL 

ct-VALUE 

Gyro 

+0.13 

+0.11 

Hydraulic  Pump 

+0.22 

+0.29 

Airborne  Navigation 

+0.15 

- 

TACAN 

+0.17 

- 

TABLE  6-4:  COMPARISON  OF  RELIABILITY  GROWTH  RATES 


TYPE  OF 

PROGRAM 

TYPICAL 

GROWTH  RATE 

PARAMETER 

(a) 

OPER.  TIME  TO 
DOUBLE  MTBF 
( T-j  MULTIPLES) 

Development 

Testing 

+0.41 

C  A 

yj  •  ~T 

RIW  In-Svc 
Operation 

+0.18 

47.0 

In-Service 

Improvement 

Prog. 

+0.15 

101.6 

In-Service 

Experience 

+0.05 

1,047,587.0 

6.3  Reliability  Growth  Test  Time:  The  test  time  required  to  grow  the 
reliability  to  the  specified  level  is  an  important  consideration  for 
determining  costs,  manpower  and  other  resources  and  is  extremely  dependent 
upon  the  growth  rate  and  initial  reliability  level. 

In  order  to  expose  latent  defects  as  quickly  as  possible,  efforts  can  be 
made  to  operate  equipment  in  on/off  cycles  while  applying  an  environment 
including  temperature  and  vibration  cycling.  High  temperature  will  accel¬ 
erate  chemical  deterioration,  while  extreme  temperature  cycling  will  pro¬ 
duce  thermal  stresses  and  expose  mechanical  weaknesses,  as  will  vibration. 
Repeated  on-off  switching  will  produce  both  transient  thermal  stresses  and 
electrical  stresses. 

Various  references  reconmend  test  times  tc  be  used  for  growth  testing. 
There  appears  to  be  conflict  with  regard  to  these  times  as  shown  in  Table 
6-5.  This  conflict  may  be  attributed  to  differences  in  the  magnitude  of 
the  reliability  numerical  requirements. 


TABLf-I  6-5:  VARIATIONS  OF  RECOMMENDED  TEST  TIMES  PRESENTED  IN  THE 

LITERATURE 


—  - - - -  "  - -  ■"  - - 

Recomnended  Test  Time 

Reference 

1.  20-50  multiples  of  the  required  MTBF  when  the 

3 

required  MTBF  is  not  greater  than  a  few 

hundred  hours  (tested  in  severe  environment) 

2.  Not  less  than  a  few  multiples  of  the  specified 

21 

MTBF 

3.  5  to  25  multiples  of  the  requiied  MTBF 

37 

4.  50  co  100  multiples  of  the  required  MTBF 

34 

5.  10  to  25  multiples  of  the  required  MTBF 

72 

6.3.1  Reliability  Growth  Test  Time  Estimation  for  a  System:  By  solving 
equation  5.5  for  time  we  have  a  convenient  equation  tor-  estimating  the  test 
time  needed  to  "grow"  a  system  from  some  initial  MTBF  to  the  required 
(instantaneous)  MTBF. 

1 

T  =  [(MTBFINSr)  (K)  d-0  (EqU*  6J) 

To  calculate  the  test  time  needed,  one  must  first  calculate  the  constant  K. 
This  is  done  by  using  equation  5.2  and  substituting  an  expected  growth  rate 
and  an  expected  initial  MTBFcum  after  some  initial  preconditioning  period 
Tp£  and  then  solving  for  K.  Experif  „e  with  previous  reliability  growth 
programs  should  provide  a  means  of  estimating  the  initial  MTBF  point. 
However,  if  experience  data  is  nrt  available,  as  a  last  resort,  the  follow¬ 
ing  general  approximations  can  be  used  for  planning  purposes. 


)  ( Equ .  6.2) 


This  provides  an  estimate  of  the  initial  reliability  and  the  length  of  time 
needed  to  stabilize  the  data  to  the  point  where  meaningful  assessments  and 
projections  can  be  made.  The  lower  and  upper  limits  on  Tp^,  per  equipment 
should  be  in  the  range  of  50  hours  and  300  hours  respectively.  Smaller 
equipments  usually  have  higher  MTBF's  and  thus  the  initial  condition  times 
calculated  from  equation  6.3  may  seem  excessive.  However,  Tpc  is  the  total 
conditioning  period  for  all  equipments  to  be  put  on  test,  and  when  it  is 
divided  among  the  equipments  that  are  going  to  be  tested,  the  initial 
conditioning  time  per  equipment  should  fall  in  the  range  given  above.  It 
is  important  to  understand  that  there  is  more  than  one  way  to  reach  the 
same  goal  MTBF  for  a  given  amount  of  test  time.  This  is  shown  in  Figure 
6.6.  Curve  1  depicts  an  equipment  with  a  lower  initial  starting  reliabi¬ 
lity  and  a  higher  yrowtri  rate  that  takes  T  hours  to  reach  its  goal  MTC^, 
Curve  2  represents  the  same  equipment  with  a  higher  initial  reliability  and 
a  smaller  growth  rate  except  with  increased  emphasis  placed  on  other  relia¬ 
bility  tasks  such  as:  derating,  higher  quality  parts,  and  better  thermal 
management,  etc. 
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FIGURE  6.6:  DIFFERENT  WAYS  OF  REACHING  THE  SAME  MTBF  GOAL 


TIME 

6.3.2  Allocating  Reliability  Growth  Test  Time  to  Subsystems:  Reference 
21  presents  a  method  of  allocating  reliability  growth  test  time  to  the  most 
critical  subsystems  in  order  to  concentrate  the  test  effort  on  the  region 
of  maximum  potential  benefit.  This  method  serves  as  a  check  to  assure  that 
cest  time  is  not  wasted  on  high  MTBF  subsystems.  An  example  best  illus¬ 
trates  this  method. 


Suppose  a  system  was  comprised  of  the  five  subsystems  shown  in  Table  6-6 
and  5000  hours  are  available  for  reliability  growth  testing. 
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TABLE  6-6:  SUBSYSTEMS  AND  THEIR  REQUIRED  MTBF'S 


v. . 

V  V  V  “J 


Subsystem 


Required  MTBF 
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SV.V.\ 

.1 


The  procedure  used  to  allocate  the  5000  hours  is  to  rank  the  subsystems  in 
order  from  the  lowest  MTBFrequired  to  the  highest  MTBFreq(<ired  and  then 
divide  the  total  test  time  available  evenly  among  each  subsystem  and  calcu¬ 
late  the  number  of  test  multiples  of  the  required  MTBF  as  shown  in  Table  6- 


TABLE  6-7:  TEST  TIME  IN  TERMS  OF  MULTIPLES  OF  THE  REQUIRED  MTBF 


S&J 

VvV-i 


J 

V.%' 
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Subsystem 


MTBF 


required 


Test  Multiples  of 
MTBFRequire/ 
1000/50  =  20 


1000/100  =  10 


y," >  ,n.1 

ir 


Testing  for  small  multiples  of  the  required  MTBF  Is  not  generally  as 
beneficial,  thus  subsystems  D  and  C  probably  should  not  undergo  reliabi¬ 
lity  growth  testing.  The  next  step  would  be  to  go  back  and  reallocate  the 
test  time  given  to  subsystems  0  and  C  in  order  to  obtain  greater  test 
multiples  of  MTBrrequired  (for  each  subsystem)  are  in  the  range  of  the 
recommended  test  times  given  in  Table  S-5.  Another  point  to  b  noted  is  that 
excessive  test  time  on  a  subsystem  may  also  be  inefficient;  therefore,  a 
reallocation  may  be  warranted  should  the  multiples  of  the  MT5^requirecj  be 
too  high. 

6.3.3  Test  Time  Example:  Suppose  the  early  part  of  a  reliability  growth 
test  generated  failure  data  as  shown  in  Table  6-8  and  one  wanted  to  make  an 
estimate  of  the  test  time  needed  to  achieve  an  MTBF  of  70  hours  using  this 
failure  data. 


TABLE  6-8:  INITIAL  GROWTH  TEST  DATA 


Cumulative  Test  Hours 

Cumulative  MTBF  (Hrs) 

200 

20 

525 

30 

980 

35 

1500 

39 

1/00 

39 

88 
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Plotted  on  log-log  paper  (Figure  6.7)  this  data  shows  that  reliability  Is 
Improving  in  a  linear  manner. 


After  a  linear  relationship  becomes  apparent,  a  straight  line  can  be  drawn 
through  the  data  points  and  the  parameters  of  the  Duane  model  can  be  calcu¬ 
lated  as  follows: 


The  practice  of  using  only  two  data  points  to  calculate  a  should  be 
avoided.  However,  it  is  done  in  this  example  because  the  two  points  used 
lie  on  the  "eyeballed"  line  in  Figure  6.7  and  because  equations  5-7  and  5-8 
dr-e  tuo  lengcny  for  this  simple  example. 


2400  Hours 


Equation  6.1  could  have  been  used  as  a  more  direct  analytical  approach. 

T  =  [(70)  (.32)  ( 1  - . 35 1/,3S  =  2095 

6.3.4  Planning  Test  Time:  Many  reliability  growth  planners  fall  into 
the  trap  of  determining  test  time  based  on  the  cumulative  MTBF  reaching  the 
predicted  MTBF.  Clarke  (Ref  42)  showed  analytically  that  there  Is  a  region 
of  "no  growth"  after  the  current  MTBF  reaches  the  predicted  MTBF.  Failures 
precipitated  during  this  period  will  likely  be  nonpattern,  noncorrectable 
ones  occurring  at  a  rate  of  the  reciprocal  of  the  predicted  (inherent) 
MTBF.  Therefore,  a  test  structured  on  the  cumulative  MTBF  reaching  the 
predicted  MTBF  would  never  be  completed. 

Koo,  in  a  1981  Westinghouse  paper  (Ref  51),  showed  how  to  manipulate  growth 
models  based  on  random  effect  and  systematic  failures  to  arrive  at  test 
times  required  to  find  a  certain  percentage  of  systematic  failures,  to 
reduce  the  hazard  rate  to  a  certain  level  or  to  ensure  that  a  certain 
number  o  systematic  failures  occur. 

6.4  The  Exponential  Law  for  the  Appearance  of  Systematic  Failures:  Green 
(Ref  3)  states  that  through  severe  environmental  test  cycles  the  appear¬ 
ance  of  systematic  failures  may  follow  an  exponential  law. 

The  general  equation  for  describing  the  appearance  of  systematic  failures 
is: 
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(Equ.  6.4) 


*  t 


.  ;..v 

fcs 


'■  'A 


'mi 


i 

V> 

# 

>> 

•  .  < 

v-a* 


A,. 

V.-. 

'  «A  .. 
■■■■:■. 

*3 


v  A* 

'  * 

>\\m 

« *  «  ■ 

•m 

,%*.N 

,N 


W.N 

«■*  * 

v  \0_'-  . 


where:  F-j-^g  =  Type?  of  systematic  failure  observed 

Fjsp  *  Types  of  systematic  failures  present 
z  =  Time  constant  for  the  environmental  test  condition 
(z  decreases  with  increasing  test  severity) 
t  =  Cumulative  operating  time 

A  time  constant  of  400  hours  has  been  observed  in  complex  airborne  radar 
systems.  This  indicates  that  on  any  one  equipment,  after  testing  for  1000 
hrs  under  a  severe  environment,  90%  of  the  systematic  defects  are  revealed 
(l.e.,  1  -  e  ("1000/400)  =  o.9).  This  is  shown  graphically  In  Figure  6.8. 

FIGURE  6.8:  EXPONENTIAL  LAW  FOR  THE  APPEARANCE  OF  SYSTEMATIC  FAILURES 
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The  optimum  test  duration  per  equipment  depends  on  the  target  MTBF,  and 
only  in  the  case  of  an  MTBF  of  several  hundred  hours  or  of  investigation 
into  long  term  wearout  failures  is  it  worth  extending  the  test  on  any  one 
equipment  beyond  1000  to  1500  hours  if  other  equipment  are  available  for 
testing. 

Green  also  states  that  in  his  experience  no  single  equipment  lias 
accumulated  more  than  3000  hours  of  operation  per  annum  following  a  burn-in 
test.  2500  hours  is  a  typical  maximum  rate  per  equipment  for  accumulating 
operating  hours. 

Reference  54  applied  the  following  criteria  in  order  to  identify 
systematic  failures  (as  opposed  to  random  failures).  If  either  of  the 
criteria  below  is  met,  a  possible  systematic  reliability  problem  was 
identified: 

A.  The  ratio  of  the  number  of  failed  parts  to  the  parts  applications 
was  greater  than,  or  equal  to  5  percent,  for  parts  population  of  greater 
than  100. 

B.  The  ratio  of  the  number  of  failed  parts  to  the  parts  application 
was  greater  than  20  percent  and  the  number  of  failures  was  greater  than  1, 
for  parts  population  of  less  than  100. 
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From  a  mathematical  aspect,  the  exponential  law  is  not  compatible  with 
Duane's  model.  By  differentiating,  it  is  apparent  that  log  of  failure  rate 
will  be  proportional  to  time,  and  not  log  time  as  is  the  case  with  the 
Duane  model.  The  reason  for  this  is  that  the  Duane  model  tracks  additional 
failures  such  as  random  failures,  quality  control  type  failures,  wear-out 
failures  and  repetitive  systematic  failures  where  the  complete  cure  lias 
not  been  found. 

For  a  high  target  MT8F  of  several  hundred  or  thousands  of  hours,  the 
limitations  <n  development  time  and  money  and  the  inability  to  use  multiple 
samples  may  preclude  extensive  growth  testing  and  accelerated  stress  test¬ 
ing  may  be  essential  for  equipment  requirements  to  be  achieved  in  a  cost 
effective  m.nner.  However,  accelerated  testing  must  be  planned  and  used 
with  cautior  so  unrealistic  failure  modes  will  not  be  revealed. 

6.5  TrcCking  Techniques:  The  basic  reasons  to  track  reliability  growth 
(or  decline)  are  to  make  assessments  of  reliability  against  the  planned 
values  and  to  project  future  ..liability. 

The  planned  reliability  growth  provides  a  standard  to  which  results  can  be 
compared.  Assessments  can  be  made  without  a  planned  reliability  growth 
curve;  however,  the  comparisc’  is  subjective  because  there  is  no  standard 
against  which  to  judge  and  it  is  a  matter-  of  opinion  whether  or  not  the 
program  is  progressing  satisfactori ly.  Further,  assessment  provides  a 
clear  indicator  to  a  program  manager  when  something  has  gone  wrong  so  he 
may  know  when  corrective  action  needs  to  be  taken. 
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Growth  assessment  should  only  be  made  after  some  settling  down  period  if  a 
development  phase  or  test  phase  change  has  just  taken  place  or  new  equip¬ 
ment  interfaces  have  been  added.  Substantial  reliability  decline  (dips) 
may  result  from  infant  mortalities  resulting  from  new  interfaces  and  from 
the  need  for  a  learning  process  at  the  start  of  a  new  phase  as  mentioned 
earlier. 

Reliability  growth  projection  is  used  after  a  trend  has  been  established. 
It  is  particularly  useful  when  the  current  estimate  of  reliability  varies 
significantly  from  the  pi  aimed  value  because  it  can  be  used  to  allot  more 
or  less  test  time  to  the  current  test  phase  or  to  intensify  the  growth 
effort  tc  stimulate  a  greater  growth  rate. 

Another  methud  that  can  be  used  to  track  reliability  and  signa1  trouble  in 
a  growth  program  is  the  Triple  Tracking  method  presented  by  Simkins  (Ref 
44).  This  method  is  a  real-time  reliability  measurement,  tracking,  and 
control  approach  that  is  implemented  during  the  development  of  a  new  sys¬ 
tem.  It  allows  for  multitier  growth  tracking  (equipment,  subsystem,  and 
system)  arid  provides  a  high  degree  of  management  visability  into  the  effec¬ 
tiveness  of  corrective  actions. 

The  basic  approach  is  to  establish  cumulative  and  instantaneous  target 
curves  using  Duane  techniques  and  then  plot  failures  as  they  occur  to 
develop  actual  cumulative  and  instantaneous  curves.  The  instantaneous 
plot  is  obtained  by  censoring  all  correctible  failures  and  not  by  jumping 
up  the  cumulative  plot  by  a  factor  of  yy,  as  is  done  with  a  Duane  plot. 
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The  cumulative  plot  is  obtained  by  plotting  all  relevant  failures.  Confi¬ 
dence  bounds  for  both  the  cumulative  and  instantaneous  plots  are  then 
calculated  using  the  chi-squared  method.  There  are  three  conditions  that 
must  exist  for  a  "red  flag"  condition  which  necessitates  major  redesign, 
major  change  in  management  control,  overhaul  or  new  negotiations  on  speci¬ 
fication  requirements.  These  out-of -tolerance  conditions,  all  of  which 
must  be  present  for  a  "red  flag"  condition  are: 

A.  Confidence  bands  about  each  best  estimate  of  the  instantaneous 
MTBF  do  not  include  the  instantaneous  targeted  curve  (planned  instan¬ 
taneous  MTBF  curve). 

B.  Confidence  bands  about  each  best  estimate  of  cumulative  MTBF  do 
not  include  the  cumulative  targeted  curve  (planned  cumulative  MTBF  curve). 

C.  The  projections  do  not  reach  the  MTBF  goal  before  the  end  of  each 
of  the  three  major  test  periods:  development,  integration  and  postinte¬ 
gration. 

If  only  one  or  two  of  the  above  conditions  exist  then  a  minor  out-of¬ 
tolerance  condition,  "yellow  flag"  condition  will  exist.  Minor  out-of¬ 
tolerance  conditions  are  those  conditions  requiring  limited  actions  such 
as  only  one  equipment  out  of  a  system  needing  redesign,  more  frequent 
design  reviews,  special  task  studies  on  pattern  problems,  or  more  direct 
subcontractor  control. 


98 


A  benefit  of  the  triple  tracking  scheme  is  that,  once  an  out-of-tolerance 
condition  exists,  the  program  manager  knows  more  about  what  might  be  the 
cause  of  it.  For  example,  if  the  projection  and  cumulative  tracking  are 
within  bonnes,  but  the  instantaneous  measurements  are  below  target,  then 
he  knows  that  not  enough  censoring,  at  least  recent  censoring,  has  taken 
place.  That  is,  not  enough  corrective  actions  have  been  found,  implemented 
aid  verified,  at  least  recently. 

Another  useful  indicator  that  can  be  used  in  tracking  reliabilty  growth  has 
beer,  observed  by  Green  (Ref  3).  He  states  that  if  the  failures  are 
classified  as  systematic  or  random,  then  the  ratio  of  systematic  to  random 
provides  a  useful  indicator  of  progress.  Initially,  tne  ratio  is  about 
5:1.  When  the  ratio  falls  to  between  1:1  and  2:1,  the  reliability  target 
has  usually  been  attained  and  by  that  time  there  is  uncertainty  i,:  the 
categorization  of  failures. 


6.6  Confidence  Levels:  Since  the  system  configuration  is  continually 
changing  during  a  reliability  growth  program,  there  is  usually  limited 
test  data  available  on  the  system  for  a  fixed  configuration.  Consequently, 
direct  estimates  of  system  reliability  for  a  fixed  configuration  would 
generally  not  enjoy  a  high  degree  of  confidence  and  may,  therefore,  have 
little  practical  value.  However,  relatively  recently  confidence  intervals 
were  presented  in  MIL-HDBK-189  for  use  with  the  AMSAA  Model. 


A  unique  method  for  calculating  confidence  intervals  for  the  Duane  model  is 
presented  by  Mead  (Ref  18).  A  "least  squares"  technique  is  used  to  fit  a 
line  to  Duane  growth  points.  As  each  successive  point  contains  more 
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TABLE  6-10:  RELIA3ILITY  ATTRIBUTE  LEVELS  FOR  A  GIVEN  STATE 
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FIGURE  6.9:  PERCENT  INCREASE  IN  ACQUISITION  COST  -  VS  -  NORMALIZED  MTBF 


MTBF,  0  (Normalized) 

Where:  9q  -  represents  the  MTBF  of  the  equipment  when  applying  the  lowest 

level  associated  with  each  reliability  attribute 


0^  -  represents  the  MTBF  of  the  equipment  when  applying  the  high¬ 
est  level  associated  with  each  reliability  attribute. 
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FIGURE  6.10:  RELIABILITY  TASK  COST  RELATIONSHIPS 
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Figure  6.10  shows  different  reliability  task  cost  relationships  on  the 
data  given  in  Table  6-10  and  their  payoffs.  As  can  be  seen,  reliability 
growth  testing  has  about  the  same  magnitude  of  cost  effectiveness  as  other 
well  accepted  reliability  program  tasks  such  as  parts  selection  and 
derating. 


There  is  reason  to  believe  this  data  may  be  pessimistic  with  respect  to  the 
cost  effectiveness  of  reliability  growth  testing  because: 


A.  The  data  represents  only  the  Federal  Aviation  Administration's 
(FAA)  findings  and  therefore  may  not  be  representative  of  the  complexity  of 
DoD  systems.  The  greater  the  complexity  of  the  system,  the  less  likely  it 
is  that  all  the  problems  will  be  found  during  the  design  phase  and  the  more 
cost  effective  growth  testing  becomes. 
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B.  RDGT  Is  a  cost  effective  complement  to,  not  substitute  for,  other 


reliability  tasks. 


m 


C.  The  systems  represented  are  likely  to  have  greater  maturity  than 
DoD  systems.  The  FAA  uses  many  more  off-the-shelf  or  modified  designs. 

7.0  Reliability  Growth  Application  Experience:  This  section  will  present 
an  overview  of  some  interesting  observations  and  unique  test  approaches 
that  have  been  found  in  the  course  of  the  study. 

7.1  Current  Air  Force  Applications:  A  number  of  Air  Force  system  program 
offices  (SPO's)  were  contacted  to  determine  where  reliability  growth 
testing  has  been  applied  or  is  being  planned.  Table  7-1  lists  the  program 
name,  the  organization  responsible  for  the  program  and  the  type  of 
system/equipment  under  development  for  the  information  gathered. 
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TABLE  7-1:  AIR  FORCE  RELIABILITY  GROWTH  APPLICATIONS 


PROGRAM  NAME 


ORGANIZATION 


HAVE  CLEAR 
(Formerly  SEEK  TALK) 

ESD 

SACDIN 

ESD 

AFSATCOM 

ESD 

JTIDS 

ESD 

Simulator  SPO 

ASD 

F-16 

ASD 

B1  -B 

ASD 

AMRAAM 

AD 

B-5?  Offensive 

Avionics  System  (OASj 

ASD 

AWACS 

ESD 

AN/ARC- 164 (V) 

ASD 

TYPE  OF  SYSTEM 


EQUIPMENT 


UHF  Radio 


Coumuni cat  ions 

Communications 

Class  II  Terminal 

Aircraft  Simulators 

Aircraft  Radar 

Different  Electronic 
Systems  and  Some 
Electro-Mechanical 
Systems 

Missile 

Various  Onboard 
Electronic  Systems 

Airborne  Surveillance 
Radar,  Data  Prucess  i ny 
Displays,  Communication 
Navigation 

Communications 


of  55  hours  to  an  MTBF  of  250  hours  at  the  start  of  low  rate  initial  produc¬ 
tion.  Reliability  growth  testing  will  continue  through  low  rate  initial 
production  with  the  final  jjal  being  550  hours. 

7.1.2  SACDIN  -  The  initial  reliability  program  included  a  Failure 
Reporting  and  Corrective  Action  System  (FRACAS)  and  a  reliability  demon¬ 
stration  test.  Past  experience  showed  that  little  attention  was  given  to 
analysis  of  failures  and  corrective  action.  Thus,  an  integrated  growth 
test  is  being  conducted  both  as  a  development  tool  and  as  a  determination 
of  contractual  compliance  with  the  required  reliability.  The  Duane  model 
is  being  used  for  planning  and  tracking  purposes. 

Thus  far,  the  results  of  this  growth  test  are  showing  a  growth  rate  of  .3 
to  .4.  Since  testing  continued  before  corrective  actions  were  taken  and 
all  failures  were  counted,  some  functional  areas  failed  to  meet  reliabi¬ 
lity  milestones. 

7.1.3  AFSATCQM  -  A  reliability  growth  assessment  was  performed  on  the 
Terminal  Segment  in  a  modification  of  a  standard  MIL-STD-781  RQT 
accept/reject  criteria.  In  order  to  use  the  contractually  required  MIL- 
STD-781  B  test  plan  for  a  combined  growth  and  demonstration  test  a  ground 
rule  was  made  which  allowed  failures  caused  by  design  deficiencies  to  be 
censored  from  the  accept/reject  count  after  the  corrective  action  design 
change  was  implemented  and  verified.  A  typical  verification  test  time  was 


used  2  or  3  times  the  specified  MTBF.  The  procedure  used  was  the  topic  of  a 
1977  Reliability  and  Maintainability  Symposium  paper  "AFSATCOM  Terminal 
Segment  Reliability  Test  Program"  (Ref.  19). 

7.1.4  JTIDS  -  The  Class  II  terminal  of  the  JTIDS  System  will  undergo  a 
period  of  reliability  growth  testing  of  between  1000  hours  and  2000  hours. 
The  exact  lengt,  of  the  growth  test  is  dependent  upon  whether  the  current 
MTBF  equals  or  exceeds  a  required  MTBF  of  500  hours.  A  formal  reliability 
demonstration  test  is  required  at  the  completion  of  growth  testing. 

7.1.5  Simulator  SPO  -  Because  of  the  small  number  of  aircraft  simula¬ 
tors  usually  procured  (10-15),  the  design  changes  are  implemented  during 
the  program  which  makes  all  systems  slightly  different.  This  factor,  along 
with  the  use  of  some  commercial  off-the-shelf  equipment,  makes  for  the  use 
of  a  reliability  growth  test  as  a  means  to  determine  contractual  compliance 
on  some  programs.  The  goal  MTBK's  are  usually  in  the  range  of  10  to  40 
hours  and  the  tests  are  performed  in  a  laboratory  environment  since  that  is 
the  usual  field  environment. 

7.1.6  F-16  -  A  dedicated  reliability  growth  test  was  performed  on  the 
first  generation  Westinghouse  radar  with  good  results.  A  new  avionics 
package  is  under  development  for  the  F-16  and  a  growth  test  is  planned  for 
the  new  radar.  Originally,  two  equipments  were  to  be  tested  for  500  hours 
each.  However,  field  experience  from  the  first  generation  radar  showed 
tl;at  most  failures  occurred  soon  cite.'  the  system  was  started.  Because  of 
this  past  experience,  it  was  decided  to  test  7  radars  107  hours  each, 
for  a  total  test  time  of  about  750  hours.  This  test  length  is  about  10 
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tlmes  the  goal  MTBF  of  70  hours.  The  growth  testing  will  be  part  of  a  full 
reliability  program. 

Field  reliability  growth  on  the  F-16  fleet  is  also  tracked  using  a  computer 
software  package.  The  program  can  track  monthly,  quarterly,  or  cumulative 
trends  in  reliability.  It  also  tracks  the  trends  of  different  work  unit 
codes  to  pinpoint  developing  problem  areas.  A  cumulative  growth  rate  of 
about  .25  has  been  observed  for  the  fleet. 

7.1.7  B1 -B  -  A  dedicated  reliability  growth  test  of  1000  to  2000  hours 
is  planned.  The  testing  will  take  place  on  complex  equipment  that  is 
either  new  or  modified.  Two  first  production  units  of  each  LRU  will  be 
tested.  Each  test  unit  must  accumulate  a  minimum  of  25%  of  the  total  test 
time  allotted  for  the  two  units.  The  Duane  Model  will  be  used  for  planning 
and  tracking  purposes  and  a  growth  rate  of  about  .3  is  expected. 

Because  of  funding  and  schedule  constraints.  Environmental  Stress  Screen¬ 
ing  and  Reliability  Growth  Testing  are  the  only  reliability  tasks  required 
and  contractual  compliance  with  reliability  will  be  determined  based  on 
their  results. 

7.1.8  AMRAAM  -  It  is  planned  that  six  missiles  will  be  put  on  test  to 
accumulate  12,500  hours  of  reliability  growth  testing.  A  seventh  missile 
has  been  allocated  to  the  test  to  replace  any  missile  that  is  undergoing 
failure  analysis.  A  conservative  10%  of  the  goal  MTBF  has  been  assumed  for 
the  initial  starting  point.  The  goal  MTBF  is  1000  hours.  An  assumed 
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growth  rate  of  .5  is  being  planned  and  up  to  18,000  hours  of  test  time  may 
be  used  if  a  lower  starting  point  or  growth  rate  is  realized. 

All  missiles  will  undergo  Environmental  Stress  Screening  and  ten  missiles 
will  be  allocated  for  a  reliability  demonstration  test  following  growth 
testing. 
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7.1.9  B-52  OAS  -  Initially,  reliability  requirements  were  minimal. 
When  additional  funding  became  available,  a  dedicated  reliability  growth 
test  of  2400  hours  per  system  was  chosen.  No  specific  growth  rate,  start¬ 
ing  point  or  target  MTBF  were  set  before  the  test  started. 

The  test  results  indicated  that  7G%  of  all  failures  occurred  within  the 
first  1200  hours  on  most  systems.  Observations  after  the  test  also  showed 
that  the  initial  MTBF  was  about  25%  of  the  final  MTBF  and  the  growth  rate 
varied  between  .3  and  .6. 

7.1.10  AWACS  -  The  AWACS  program  used  all  test  data  (laboratory,  flight 
line,  flight)  to  evaluate  reliability  growth  using  the  Duane  concept.  The 
following  types  of  equipment  were  tested:  data  processing,  display,  iden¬ 
tification,  navigation  and  communication.  A  brassboarc!  program  was  imple¬ 
mented  which  involved  a  competitive  flyoff  of  two  prototype  surveillance 
radars,  each  installed  in  a  Boeing  707  aircraft  test  bed,  followed  by 
evaluation  and  selection  of  a  winner.  The  competitive  nature  of  the 
brassboard  phase  produced  intensive  efforts  by  both  competing  companies  to 
quickly  identify  and  eliminate  the  cause  of  failure  problems.  In  addition, 
reliability  growth  testing  was  used  as  demonstration  of  contractual 
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requirements.  The  equipment  was  accepted  if  the  slope  of  the  current 
system  MTBF  was  positive  and  the  current  system  MTBF  was  at  least  the 
specified  level  at  any  time  after  the  first  500  valid  flight  hours  (about 
12  MTBF's).  The  AWACS  reliability  growth  program  was  the  subject  of  the 
19/5  Reliability  and  Maintainabil ity  Symposium  paper  entitled  "Reliability 
Developments  -  AWACS"  (Ref  7). 


7.1.11  AN/ARC- 164  -  The  radios  accumulated  a  total  of  10,135  valid  test 
hours  with  16  relevant  failures  occurring  during  this  time.  MTBF  growth 
data  was  presented  weekly  throughout  the  test  to  provide  some  indication  of 
how  well  the  sys  ms  were  doing.  Initial  reliability  was  about  32  percent 
of  the  final  reliability  after  a  period  of  250  hours  per  system.  A  growth 
rate  between  .32  and  .35  was  realized  during  the  test. 


7 . 2  Program  Application  Summary 

From  the  proceeding  program  highlights,  it  can  be  seen  that  reliability 
growth  testing  has  been  and  is  being  applied  to  a  wide  variety  of  systems 
under  development.  Each  growth  test  was  tailored  (or  is  being  planned)  to 
meet  the  specific  constraints  of  the  overall  development  program. 
Scheduling,  funding,  the  number  of  systems  being  built  and  the  complexity 
of  the  system  seem  to  be  the  deciding  factors  on  what  type  of  testing  wil  I 
take  place  and  whether  growth  testing  will  substitute  any  other  reliabi¬ 
lity  tasks. 


8.0  CONCLUSIONS:  The  impact  of  equipment  and  system  reliability  on  oper¬ 
ational  readiness  and  life  cycle  costs  is  tremendous.  The  cost  effective 
development  of  reliable  equipment  for  the  Air  Force  is  an  important  respon¬ 
sibility.  While  the  complexity  of  today's  electronic  systems  makes  it 
virtually  impossible  to  assure  high  reliability  based  on  th°  results  of  a 
"drawing  board"  design,  some  elements  of  the  Air  Force  have  been  hesitant 
to  apply  reliability  growth  techniques. 

The  reliability  achieved  on  previous  military  systems  has  been  highly 
dependent  on  the  emphasis  placed  on  reliability  by  program  management. 
This  so-called  reliability  implementation  has  been  referred  to  as  "ad  hoc" 
depending  on  the  strength  of  the  program  office  reliability  engineer.  Use 
of  a  reliability  growth  approach  gives  the  status  of  the  reliability  pro¬ 
gram  more  visibility  and  provides  the  program  manager  with  a  toe'  for 
planning,  tracking  and  projecting.  Current  Air  Force  directives  and  regu¬ 
lations  require  that  program  managers  track  and  manage  the  reliability 
growth  process.  Earlier  revisions  assumed  that  the  specified  reliability 
could  be  designed  into  the  equipment.  Many  programs  reached  an  "accept" 
MIL-STD-781  decision  only  after  several  restarts.  Although  the  unsuccess¬ 
ful  attempts  weren't  called  "growth  testing,"  +hat's  what  they  amounted 
to.  Like  a  growth  test,  the  equipment  reliability  improved  by  design  or 
manufacturing  changes.  While  reliability  growth  can  and  does  occur  in  all 
program  phases  (i.e.,  development,  production,  and  initial  operation),  it 
is  clear  that  the  cost  effectiveness  of  the  process  becomes  greatly  dimin¬ 
ished  the  later  the  process  takes  place.  By  the  same  token,  all  potential 
problems  cannot  be  surfaced  during  DT&E,  so  full  growth  can't  be  expected 
in  development.  With  the  start  of  each  new  phase,  changes  in  manufacturing 
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processes  and  workers  introduce  a  temporary  reliability  degradation.  The 
RIW  (Reliability  Improvement  Warranty)  is  a  means  of  continuing  the  growth 
process  into  the  initial  deployment  phase. 

It  should  be  recognized  that  reliability  growth  testing  is  not  a  panacea 
for  developing  a  reliable  product.  It  is  also  not  a  substitute  for  other 
reliability  engineering  tasks  such  as  parts  control  and  stress  derating. 
Swett  (Ref.  25)  several  years  ago  likened  reliability  development  testing 
to  the  linebackers  on  a  football  team  with  the  design  phase  as  the  defen¬ 
sive  line.  Both  elements  are  necessary  for  success  whether  in  football  or 
reliability  design.  The  key  is  to  "nail  the  potential  reliability  problems 
as  early  as  possible."  A  multitude  of  cases  of  misunderstanding  and 
misapplication  of  the  growth  testing  concept  could  be  cited  where  after  the 
fact  data  has  been  used  to  show  a  growth  success  story,  or  as  Clarke 
stated,  there  was  a  "no-growth  growth"  process.  With  the  complexity  of 
today's  electronic  equipment,  it  is  impossible  to  catch  all  reliability 
problems  with  the  defensive  line. 

While  it  is  generally  agreed  that  some  sort  of  RDGT  is  needed  as  part  of 
the  development  process  many  questions  remain  regarding  implementation  of 
the  concept.  Table  8-1  (Previously  presented  in  Section  2.1)  lists  many  of 
the  questions  often  expressed  by  those  skeptical  of  RDGT. 


TABLE  8-1:  QUESTIONS  REGARDING  RDGT  IMPLEMENTATION 


1.  Who  pays  for  the  RDGT?  Does  the  government. end  up  paying  more? 

2.  Does  RDGT  allow  contractors  to  "get  away  with"  a  sloppy  initial 
design  because  they  can  fix  it  later  at  the  government's  expense? 

3.  Should  reliability  growth  testing  be  dedicated  or  integrated? 

4.  When  should  a  reliability  growth  test  begin? 

5.  Should  reliability  growth  be  planned  for  beyond  the  FSED  phase? 

6.  Should  the  equipment  operate  at  the  fully  specified  performance 
level  prior  to  the  start  of  RDGT? 

7.  Should  all  development  programs  have  some  sort  of  reliability 
growth  testing? 

8.  How  does  the  applicability  of  reliability  growth  testing  v»ry 
with  the  following  points  of  a  development  program? 

a.  Complexity  of  equipment  and  its  challenge  to  the  state-of- 

the-art 

b.  Operational  environment 

c.  Quantity  of  equipment  to  be  produced 

9.  What  model (s)  should  be  used? 

10.  What  starting  points  and  growth  rates  should  be  used  for 
planning? 

11.  How  much  test  time  will  be  required? 

12.  When  will  corrective  actions  be  implemented  and  how  will  failures 
be  counted? 

13.  Will  there  ot  u.-;  cccept/reject  criteria? 

14.  Should  the  contractor  be  responsible  for  intermediate  milestones? 

15.  Can/should  growth  testing  be  incentivized? 

16.  Does  the  type  of  contract  affect  RDGT  decisions? 

17.  What  is  adequate  time  for  verifying  a  design  fix? 

18.  What  is  the  relationship  between  an  RQT  and  RDGT? 

19.  Whu  will  do  the  trucking?  How  and  to  whom  will  the  results/status 
be  reported? 

20.  How  much  validity/confid-mce  =houlu  be  placed  on  the  numerical 
results  of  RDGT? 


Based  on  the  findings  of  the  study,  the  following  paragraphs  will  address 
each  of  these  questions: 
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1.  Who  pays  for  the  RDGT?  Does  the  government  end  up  paying  more? 

The  usual  case  is  that  the  government  pays  for  the  RDGT  as  an  additional 
reliability  program  cost  and  in  stretching  out  the  schedule.  There  have 
been  situations  where  contractors  have  tested  on  their  own  prior  to  an  RQT 
as  their  means  of  reducing  the  risk  of  an  RQT  reject  decision.  In  a 
competitive  environment,  usually  the  offeror's  will  riot  risk  losing  the 
contract  because  of  the  extra  cost  of  nonrequired  growth  testing.  The 
point  to  be  made  with  regard  to  the  RDGT  cost  is  that,  regardless  of  who 
pays,  the  reliability  will  be  improved  and  the  support  cost  element  of  the 
total  life  cycle  cost  equation  will  be  reduced.  The  savings  in  support 
costs  (recurring  logistics  costs)  exceed  the  additional  initial  acquisi¬ 
tion  cost,  resulting  in  a  net  savings  in  LCC.  The  amount  of  these  savings 
is  dependent  on  the  quantity  to  Le  fielded,  the  maintenance  concept,  the 
sensitivity  of  LCC  to  reliability  and  the  level  of  development  required. 
It  is  the  old  "pay  rne  now  or  pay  me  later  situation"  which  in  many  cases 
makes  a  program  manager's  situation  difficult  because  his  performance  is 
mainly  based  on  the  "now"  performance  of  cost  and  schedule.  Figure  8.1 
show:  how  ‘  3  extra  jevelopme"*  cost  of  an  RDGT  is  "paid  back"  by  reduced 
-.a  cvrie  coste 
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reliability  (resulting  from  an  inadequate  contractor  design  process)  will 
necessitate  an  unrealistic  growth  rate  in  order  to  attain  an  acceptable 
level  of  reliability  in  the  allocated  amount  of  test  time.  The  growth  test 
should  be  considered  as  an  organized  search  and  correction  system  for 
reliability  problems  that  allows  problems  to  be  fixed  when  it  is  least 
expensive.  It  is  oriented  towards  the  efficient  determination  of  correc¬ 
tive  action.  Solutions  are  emphasized  rather  than  excuses.  It  can  give  a 
nontechnical  person  an  appreciation  of  reliability  and  a  way  to  measure  its 
status. 

3.  Should  the  RDGT  be  dedicated  or  integrated? 

The  decision  regarding  whether  to  allot  a  specified  number  of  hours  for  a 
dedicated  growth  test  has  many  pro's  and  con's.  Dedicated  tests  have  the 
following  advantages: 

A.  Better  control  is  maintained  wit’'  respect  to  failure  occurrence, 
documentation  and  reporting. 

B.  There  is  less  chance  of  inducement  of  failures  by  operators,  test 
equipment,  etc. 

C.  The  environmental  conditions  are  easier  to  control. 

D.  Use  of  the  resulting  data  for  assessment  and  projection  has 
greater  validity. 
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E.  The  equipment  usually  has  a  pre-established  baseline  performance 
(including  meeting  environmental  qualification)  against  which  to  judge 
failures. 


F.  The  equipment  more  closely  represents  the  configuration  and  manu¬ 
facturing  processes  to  be  used  in  production. 


On  the  other  side  of  the  coin  are  the  following  arguments  for  an  integrated 
RDGT : 


A.  Since  a  separate  period  of  testing  is  not  required,  the  cost  is 
obviously  less. 


B.  This  form  of  testing  is  more  in  line  with  the  cost  effective 
spirit  of  RDG1  via  earlier  detection  and  correction  of  failures. 


The  attributes  of  dedicated  and  integrated  testing  change  when  an  attempt 
is  made  to  use  the  testing  as  a  determination  of  contractual  compliance 
with  numerical  requirements.  Reliability  problems  should  be  uncovered  and 
corrected  as  early  as  possible  to  be  most  cost  effective.  As  pointed  out 
earlier,  an  RDGT  implies  more  structured  planning,  assessment  and  tracking 
than  TAAF  and  FRACAS.  As  such,  a  performance  baseline  needs  to  be  estab¬ 
lished  prior  to  the  start  cf  the  RDGT  which  implies  a  later  start  and  a 
dedicated  test.  Integrated  tests  may  be  more  appropriate  for  small  quan¬ 
tity  very  complex  systems  and  ones  with  very  limited  test  resources. 
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Waiting  for  a  well  controlled  dedicated  test  time  with  the  equipment  per¬ 
forming  to  full  specified  capacity  will  in  most  cases  be  less  cost  effec¬ 
tive  in  providing  a  vehicle  for  correction  of  deficiencies;  however,  it 
offers  a  better  vehicle  for  assessment  and  projection.  Carrying  this  to  an 
extreme,  to  count  on  reliability  growth  later  in  the  equipment  life  cycle 
from  the  development  phase  will  be  very  cost  ineffective  due  to  the  diffi¬ 
culty  in  incorporating  design  changes. 

.  When  should  a  reliability  growth  test  begin? 


This  is  partially  answered  by  question  number  3  regarding  a  dedicated  vs  an 
integrated  RDGT  approach.  It  should  be  obvious  that  the  earlier  a  problem 
is  found  and  analyzed,  the  less  costly  it  is  to  implement  a  corrective 
n  change.  Of  course,  if  too  early,  it  is  difficult  to  determine 
whe  er  the  problem  uncovered  is  a  reliability  problem  or  a  question  of  the 
design  not  yet  meeting  the  specified  performance  criteria.  The  definition 
of  rel :  bility  reflects  the  ability  to  perform  to  some  specified  criteria 
over  e.  Therefore,  tracking  of  growth  can  only  really  be  done  after  the 
equipment  performs  at  or  near  its  specified  levels.  This  is  not  to  say 
that  uncovered  reliability  problems  should  not  be  corrected  as  early  as 
possible.  It  has  been  said  that  growth  occurs  up  to  two  years  after  IOC 
but  this  includes  growth  processes  involving  the  human  element. 
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5.  Should  reliability  growth  be  planned  for  beyond  the  FSED  phase? 
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As  several  authors  have  mentioned  (as  referenced  in  earlier  sections), 
there  are  different  types  of  reliability  growth  in  the  general  sense.  Our 


discussions  have  been  purposely  limited  to  the  strict  definition  of  relia¬ 
bility  growth  to  include  only  reliability  improvement  as  the  result  of 
finding,  analyzing  and  implementing  design  corrections  for  reliability 
problems  uncovered  during  testing.  In  this  sense,  the  cost  of  incorporat¬ 
ing  design  changes  past  FSED  (Reliability  Definition  and  Dt„  -nstration 
Phase)  may  oe  prohibitive  in  terms  of  ECP's  and  possible  retrofit.  The 
cost  effectiveness  of  reliability  growth  varies  inversely  with  the  program 
phase.  Therefore,  this  type  of  reliability  testing  should  be  used  in  FSED. 
Of  course,  exceptions  to  tfrs  rule  have  occurred  in  the  past  and  will 
continue  to  occur.  Cases  in  point  are  usually  the  result  of  poor  field 
reliability,  where,  as  the  result  of  an  LCC  analysis,  it  becomes  cost 
effective  to  undertake  some  sort  of  reliability  improvement  program. 
Other  situations  where  the  growth  approach  may  be  appropriate  are  Low  Rate 
Initial  Production  (LRIP)  programs.  While  determined  to  be  cost  effective 
at  that  point,  it  would  have  been  much  more  cost  effective  to  find  those 
problems  and  correct  them  during  development.  Reliability  Improvement 
Warranty  (RIW)  efforts  can  be  thought  of  as  reliability  growth  the 
production  phase.  These  efforts  aren't  always  effective  if  a  contractor 
determines  he  can  make  a  profit  without  higher  reliability  because  of 
inexpensive  maintenance.  Other  forms  of  growth  as  expressed  in  the  pre¬ 
viously  mentioned  corrments  on  "no-growth  growth"  and  "endless  burn-in" 
will  occur  in  production  and  operational  use  but  are  not  appropriate  for 
development. 

6.  Should  the  equipment  operate  at  the  fully  specified  performance  level 
prior  to  the  start  of  RDGT? 


119 


Waiting  until  every  specified  parameter  is  met  is  wasting  valuable  test, 
analysis,  corrective  action  and  verification  time.  But  on  the  other  hand, 
the  ability  to  determine  "when  is  a  failure  a  failure"  without  a  defined 
baseline  is  difficult.  If  an  equipment  is  performing  "almost"  to  specifi¬ 
cation,  determination  can  be  made  with  respect  to  most  problems  as  to 
whether  they  are  performance  related  or  reliability  related.  Because  this 
is  the  case,  the  time  to  start  is  when  any  meaningful  equipment  level 
reliability  data  can  be  developed  with  respect  to  acceptable  measures  of 
performance.  In  other  words,  if  a  radar  is  not  fully  meeting  it's  specifi¬ 
cation  with  respect  to  range,  that  should  not  prevent  test,  analysis  and 
implementation  of  corrective  design  on  the  power  supply,  signal  processor 
or  other  functional  elements.  Of  course,  this  will  result  in  exposure  to 
risk  because  a  performance  design  fix  could  introduce  reliability  prob¬ 
lems.  If  the  growth  is  to  be  used  as  an  assessment  and  projection  vehicle, 
then  the  configuration  should  meet  all  performance  requirements. 

7.  Should  all  development  programs  have  some  sort  of  growth  programs? 

The  answer  to  this  question  is  yes  in  that  all  programs  should  analyze  and 
correct  failures  when  they  occur  in  prequalification  testing.  A  distinc¬ 
tion  should  be  in  the  level  of  formality  of  the  growth  program.  The  less 
challenge  there  is  to  the  state-of-the-art,  the  less  formal  (or  rigorous)  a 
reliability  growth  program  should  be.  An  extreme  example  would  be  the  case 
of  procuring  off-the-shelf  equipment  to  be  part  of  a  military  system.  In 
this  situation,  which  really  isn't  a  development,  design  flexibility  to 
correct  reliability  problems  is  mainly  constrained  to  newly  developed 
in 'erf aces  between  the  "boxes"  making  up  the  system.  A  rigorous  growth 
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program  would  be  inappropriate  but  FRACAS  should  still  be  implemented.  The 
other  extreme  is  a  developmental  program  applying  technology  that  chal¬ 
lenges  the  state-of-the-art.  In  this  situation  a  much  greater  amount  of 
design  , 1 ex i b i 1 i ty  to  correct  unforeseen  problems  exists.  Because  the 
technology  is  so  new  and  challenging,  it  can  be  expected  that  a  greater 
number  of  unforeseen  problems  will  be  surfaced  by  growth  testing.  All 
programs  can  benefit  from  testing  to  find  reliability  problems  and  cor¬ 
recting  them  prior  to  deployment,  but  the  number  of  problems  likely  to  be 
corrected  and  the  cost  effectiveness  of  fixing  them  is  greater  for  designs 
which  are  more  complex  and  challenging  to  the  state-of-the-art. 


8.  How  does  the  applicability  of  reliability  growth  testing  ''ary  with 
the  following  points  of  a  development  program? 


A.  Complexity  of  equipment?  And  challenge  to  state-of-the-art? 

The  more  complex  or  challenging  the  equipment  design  is,  the  more  likely 
there  will  be  unforeseen  reliability  problems  which  can  be  surfaced  by  a 
growth  program.  However,  depending  on  the  operational  scenario,  the  num¬ 
ber  of  equipments  to  be  deployed  and  the  maintenance  concept,  there  may  be 
a  high  LCC  payoff  in  using  a  reliability  growth  program  to  fine  tune  a 
relatively  simple  design  to  maximize  its  reliability.  This  would  apply  in 
situations  where  the  equipments  have  extremely  high  usage  rates  and  LCC 
highly  sensitive  to  MTBF. 


B.  Operational  environment? 


All  other  factors  being  equal,  the  more  severe  the  environment,  the  higher 
the  payoff  from  growth  testing.  This  is  because  severe  environments  are 
more  likely  to  inflict  unforeseen  stress  associated  reliability  problems 
that  need  to  be  corrected. 


C.  Quantity  of  equipment  to  be  produced? 


The  greater  the  quantities  of  equipment,  the  more  impact  on  LCC  by  reliabi¬ 
lity  improvement  through  a  reliability  growth  effort. 


9.  What  reliability  growth  model (s)  should  be  used? 

The  model  to  be  used,  as  MIL-HDBK-18l  says,  is  the  simplest  one  that  does 
the  job.  Section  5  went  into  detail  on  what  models  apply  best  for  a 
variety  of  situations.  Certainly,  the  Duane  is  most  common,  probably  with 
the  AMSAA  second.  They  both  have  advantages;  the  Duane  being  simple  with 
parameters  having  an  easily  recognizable  physical  interpretation,  and  the 
AMSAA  having  rigorous  statistical  procedures  associated  with  it.  MIL-STD- 
189  suggests  the  Duane  for  planning  and  the  AMSAA  for  assessment  and 
tracking.  When  an  RQT  is  required,  the  RDGT  should  be  planned  and  tracked 
using  the  Duane  model;  otherwise,  the  AMSAA  model  is  recommended  for  track¬ 
ing  because  it  allows  for  the  calculation  of  confidence  limits  around  the 
data. 


10.  What  starting  points  and  growth  rates  should  be  used  for  planning? 
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For  planning  an  RDGT,  growth  rates  and  starting  periods  should  be  based  on 
experience  with  the  development  of  similar  systems.  Rules  of  thumb,  such 
as  a  starting  point  of  1C%  of  the  inherent  (predicted)  MTBF  at  a  test  time 
of  one  half  tne  inherent  MTBF  and  a  growth  rate  of  0.4  or  0.5,  have  been 
suggested.  Growth  is  not  a  naturally  occurring  process  but  rather  takes 
place  when  failure  modes/mechanisms  are  systematically  removed.  There¬ 
fore,  it  is  always  better  to  use  historical  data  based  on  the  experience  of 
the  particular  contractor  on  similar  programs.  As  a  planning  tool,  RADC 
has  currently  underway  (reference  Section  6.2.2)  a  research  effort  that 
will  provide  guidance  regarding  the  characteristics  to  be  expected  on  a 
particular  reliability  growth  program  based  on  both  equipment  characteris¬ 
tics  and  program  attributes. 

11.  How  much  test  time  will  be  required? 

The  test  time  required,  as  shown  previously,  is  a  function  of  the  initial 
level  of  reliability  as  well  as  the  growth  rate.  Appendix  A  gives  tables 
for  various  final  target  MTBF's.  The  literature  is  rather  confusing 
regarding  growth  test  time  recommendations  as  shown  in  Table  6-5.  Because 
of  the  rates  at  which  systematic  defects  are  likely  to  occur  and  potential 
wearout  mechanisms,  test  planning  must  also  address  test  time  on  a  per- 
equipment  basis.  Test  efficiency  is  a  driver  in  determining  how  much 
calendar  time  will  be  required  to  accumulate  the  required  test  hours.  All 
these  factors  make  a  fixed  time  reliability  growth  test  the  best  choice  for 
planning  and  for  costing  by  a  contractor  in  a  competitive  situation.  Vari¬ 
ous  persons  have  suggested  accelerating  the  test  by  way  of  more  severe 
stress  levels  as  a  means  of  shortening  the  time;  however,  extreme  caution 


must,  be  exercised  so  that  new  failure  modes  aren't  introduced  that  wouldn't 
occur  in  the  operational  environment.  Some  authors  have  described  associ¬ 
ating  an  acceleration  factor  during  growth  testing  as  a  "black  art." 

12.  When  should  corrective  actions  be  implemented? 

Ideally  the  corrective  actions  should  be  put  in  right  after  the  discovery 
of  the  problem  so  that  the  growth  process  is  continuous  and  the  verifica¬ 
tion  time  for  each  fix  is  maximized.  In  this  situation  plotted  data  would 
be  smooth.  To  carry  this  out  in  practice  would  mean  tying  up  test 
resources  until  a  fix  is  found  for  every  failure,  which  cannot  be  done  in 
real  life.  The  AMSAA  (MIL-HDBK-189)  approach  is  to  use  a  phase-by-phase 
process  where  fixes  are  implemented  at  the  end  of  each  test  phase  so  that 
within  phases  the  growth  is  continuous  and  between  phases  there  are  relia¬ 
bility  "jumps."  The  problem  with  this  approach  is  that  there  isn't  any  way 
of  judging  how  large  (or  small)  the  jumps  will  be.  Several  authors  advo¬ 
cate  plotting  only  "failure  sources,"  or  first  time  failure  occurrences, 
during  growth  tracking.  With  this  approach,  further  incidents  of  these 
modes,  following  the  first  occurrence,  are  not  counted  as  long  as  a  correc¬ 
tive  action  is  implemented  with  adequate  verification  time  prior  to  test 
completion.  Others  keep  track  of  the  progress  both  ways,  "culled"  data  and 
all  data.  The  mathematics  of  models  show  that  the  growth  process  is  a  self 
purging  one  where  the  model  itself  takes  care  of  eliminating  earlier 
failures. 

Will  there  be  an  accept/reject  criteria? 


The  purpose  of  reliability  growth  testing  is  to  u, cover  failures  and  take 
corrective  actions  to  prevent  their  recurrence.  Having  an  accept/reject 
criteria  is  a  negative  contractor  incentive  towards  this  purpose.  Moni¬ 
toring  the  contractors  progress  and  loosely  defined  thresholds  are  needed 
but  placing  ac^ept/reject  criteria,  or  using  a  growth  test  as  a  iemonstra- 
tion,  defeat  the  purpose  of  running  them. 

14.  Should  the  contractor  be  responsible  for  intermediate  milestones? 

A  degree  of  progress  monitoring  is  necessary  even  when  the  contractor  knows 
that  following  the  reliability  growth  test  he  will  be  held  accountable  by  a 
final  RQT.  Tight  thresholds  make  the  test  an  RQT  in  disguise.  General 
guidance  for  determining  the  acceptability  of  progress  is  expressed  in 
MIL-S  '0-1635  (reference  Section  6.5}  and  in  the  IBM  triple  tracking 
method.  It  must  be  remembered  what  the  purpose  of  the  test  is;  there 
should  be  no  incentives  for  contractors  to  hide  failures. 

15.  Can/should  growth  testing  be  incenti vized? 

Reliability  growth  can  be  incentivized  but  shouldn't  be.  To  reward  a 
contractor  for  meeting  a  certain  threshold  in  a  shorter  time  or  by  indicat¬ 
ing  "if  the  RDGT  results  are  good,  the  RQT  will  be  waived,"  the  contrac¬ 
tor's  incentive  to  "find  and  fix"  is  diminished.  The  growth  test's  primary 
purpose  is  to  improve  the  design,  not  to  evaluate  the  design. 

16.  Does  the  type  of  contract  affect  RDGT  decisions? 


The  type  of  development  contract  is  a  procurement  strategy  decision  and  is 
usually  determined  as  a  function  of  program  risks.  Development  contracts 
are  generally  a  "cost  plus"  type  which  may  or  may  not  include  incentives. 
Production  contracts  which  are  much  easier  to  price,  because  costs  can  be 
defined,  are  usually  some  form  of  "fixed  price"  ones.  It  has  already  been 
stated  that  contracts  with  incentives  based  on  reliability  growth  give 
contractors  a  reason  to  hide  failures,  which  is  counterproductive.  If 
fixed  length  reliability  growth  testing  is  used,  it  really  doesn't  matter 
what  the  contract  type  is  because  the  test  can  easily  be  priced,  even  as  a 
separately  priced  contract  it-m. 

17.  What  is  adequate  time  to  verify  a  design  fix? 

Most  persons  agree  that  the  verification  time  to  prove  that  a  design  fix 
has  eliminated  a  particular  failure  mode  depends  on  what  the  mode  is,  what 
the  fix  is  and  how  the  fix  interacts  with  the  rest  of  the  design.  It  must 
be  long  enough  to  assure  that,  even  though  the  original  problem  has  been 
corrected,  new  time  dependent  failure  modes  haven't  been  introduced  by  the 
fix.  A  good  rule  of  thumb  is  that  the  time  should  be  at  ieast  one  MTBF 
(predicted) . 

18.  What  is  the  relationship  between  an  RQT  and  RDGT? 

The  RQT  is  an  "accounting  task"  used  to  measure  the  reliability  of  a  fixed 
design  configuration.  It  has  the  benefit  of  holding  the  contractor  ac¬ 
countable  some  day  down  the  road  from  his  initial  design  process.  As,  such, 
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he  is  encouraged  to  seriously  carry  out  the  other  design  related  reliabi¬ 
lity  tasks.  The  RDGT  is  an  "engineering  task"  designed  to  improve  the 
design  reliability.  It  recognizes  that  the  drawing  board  design  of  a 
complex  system  cannot  be  perfect  from  a  reliability  point  of  view  and 
allocates  the  necessary  time  to  fine  tune  the  design  by  finding  problems 
and  designing  them  out.  Monitoring,  tracking  and  assessing  the  resulting 
data  gives  insight  into  the  efficiency  of  the  process  and  provides  nonreli¬ 
ability  persons  with  a  tool  for  evaluating  the  development's  reliability 
status  and  for  reallocating  resources  when  necessary.  The  forms  of  testing 
serve  very  different  purposes  and  complement  each  other  in  development  of 
systems  and  equipments.  An  RDGT  is  not  a  substitute  for  an  RQT,  or  other 
reliability  design  tasks. 

19-  Who  will  do  the  tracking?  How  and  when  will  the  results/status  be 
reported? 


When  an  RDGT  is  invoked  in  conjunction  with  an  RQT,  as  recomnended,  the 
close  monitoring  of  contractor  results  isn't  as  critical  as  when  only  an 
RDGT  is  being  required.  If  an  RQT  is  providing  the  accountability  at  some 
later  time,  the  RDGT  can  be  thought  of  as  a  means  of  increasing  the  chances 
of  passing  the  RQT.  Of  course,  as  has  not  been  the  case  in  many  RQT's  in 
the  past,  the  procuring  activity  has  to  exercise  its  redesign  options 
should  a  reject  decision  be  reached  in  RQT.  Still,  wich  an  RQT  hanging 
over  his  head,  a  contractor  may  still  shortcut  his  reliability  design 
approaches  hoping  to  pass  the  RQT  by  the  usual  practices  of  declaring 
failures  nonrelevant,  induced  by  test  equipment  and  the  like.  Therefore, 
the  growth  process  should  always  be  monitored  by  the  AF  program  office. 


with  the  degree  of  scrutiny  dependent  on  how  the  results  are  to  be  used. 
Reporting  of  the  results  and  status  is  not  clearly  defined  under  present 
reliability  standards  and  data  item  descriptions  (DID's).  No  specific 
DID's  exist  for  reliability  growth.  Existing  ones  written  for  the  RQT  must 
be  tailored  for  this  application. 

20.  How  much  val idity/condifence  should  be  placed  on  the  numerical 
results  of  ROGT? 

Associating  a  hard  reliability  estimate  from  a  growth  process,  while  math¬ 
ematically  practical,  has  the  tone  of  an  assessment  process  rather  t.ian  an 
improvement  process,  esoecially  if  an  RQT  assessment  will  not  follow  the 
RDGT.  In  an  ideal  situation,  where  contractors  are  not  driven  by  profit 
motives,  a  reliability  growth  test  could  serve  as  an  improvement  and 
assessment  vehicle.  Since  this  is  not  the  real  world,  the  best  that  can  be 
done  if  meaningful  quantitative  results  are  needed  without  an  RQT,  is  to 
closely  monitor  the  contractor  RDGT.  Use  of  the  AMSAA  model  provides  the 
necessary  statistical  procedures  for  associating  confidence  levels  with 
reliability  results.  In  doing  so,  closer  control  over  the  operating  condi¬ 
tions  and  failure  determinations  of  the  RDGT  must  be  exercised  than  if  the 
test  is  for  improvement  purposes  only.  A  better  approach  is  to  use  a  less 
closely  controlled  growth  test  as  an  improvement  technique  (or  a  struc¬ 
tured  extension  of  FRACAS,  with  greater  emphasis  on  corrective  action)  to 
fine  tune  the  design  as  insurance  of  an  accept  decision  in  an  RQT.  With 
this  approach,  monitoring  an  improvement  trend  is  more  appropriate  than 
development  of  hard  reliability  estimates.  Then  use  a  closely  controlled 
RQT  to  determine  acceptance  and  predict  operational  results. 


8.1  Summary  of  Conclusions:  Certainly  no  one  in  the  development  business 
can  argue  against  uncovering  problems  and  correcting  them.  The  RDGT  has 
been  proved  to  be  an  organized  approach  to  doing  just  that.  It  does  not 
replace  other  design  oriented  reliability  tasks.  It  may  add  to  the  acqui¬ 
sition  cost  of  a  system,  but  the  reduced  risk  of  failing  a  reliability 
demonstration  and  the  reduction  in  operation  and  support  costs  more  than 
offset  this.  Most  skeptical  comments  regarding  the  growth  concept  have 
their  origin  in  situations  where  growth  techniques  have  been  misapplied  or 
used  as  a  panacea  trying  to  bail  out  a  poor  design.  When  applied  properly 
and  not  substituted  for  an  RQT,  an  RDGT  is  an  extremely  cost  effective  task 
in  the  development  process.  Unfortunately,  many  success  stories  written 
around  the  concept  are  of  the  misapplication  type  which  have  resulted  in 
•'turning-off"  reliability  practioners  to  the  concept.  The  complexity  of 
today's  equipment  necessitates  recognition  of  the  fact  that  designs  cannot 
be  perfect  off  the  drawing  board.  As  such,  a  properly  defined  and  managed 
reliability  growth  program  is  a  must  for  today's  development  efforts. 
RADC's  new  R&D  study  "Reliability  Growth  Prediction"  will  serve  as  an 
excellent  complement  to  MIL-HDBK-189  and  MIL-STD-1635  in  assuring  that  the 
concept  is  properly  applied. 


The  following  tables  contain  estimated  test  times  calculated  by  using 
equations  6.1,  6.2,  6.3  and  5.2.  These  times  are  the  number  of  hours 
needed  for  the  instantaneous  Duane  plot  to  reach  the  MTBF  goal  (9q).  The 
predicte  1  MTBF  and  che  initial  conditioning  period  (assuming  Equ.  6.3 
holds)  are  given  at  the  top  of  each  page.  The  starting  MTBF,  stated  as  a 
percentage  of  the  predicted  MTBF,  is  varied  in  increments  of  5  percent 
across  the  top  of  each  table.  The  growth  rate  (a)  is  varied  along  the  left 
side  of  the  table  in  increments  of  .05.  The  blank  spaces  in  the  table 
represent  test  time  results  which  are  less  than  five  times  the  predicted 
MTBF  and  are  therefore  not  recommended.  A  minimum  test  time  of  5  times  the 
predicted  MTBF  should  be  used  in  these  cases. 


TA8LE  A. 2:  RELIABILITY  GROWTH  TEST  TIME  (0n  =  50  HOURS) 
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RA VC  plan*  and  execute*  reAearch,  do.vttopm2.nt,  te*t  and 
a elected  acqulAltlon  program*  In  a upport  of  Command,  Contact 
Communication*  and  Intelligence  (C3 1)  actlvltle*.  Technical 
and  engineering  *  upport  within  area*  of  technical  competence 
fj>  provided  to  ESV  Program  Office*  (POa)  and  other  ESD 
element* .  The  principal  technical  truAAlon  area*  are 
communication *,  electromagnetic  guidance  and  control,  *ur~ 
velllance  of  ground  and  aeUApace  object A,  Intelligence  data 
collection  and.  handling,  Information  *y*tem  technology , 
lonoApherlc  propagation,  Aotld  a  tote  AclenceA,  microwave 
phyAlcA  and  electronic  reliability,  maintainability  and 
compatibility. 


